ONNX Runtime & CoreML May Silently Convert Models to FP16

Deploying machine learning models to production is a complex dance of optimization, compatibility, and performance. Frameworks like ONNX Runtime and Apple's CoreML are indispensable tools in this process, promising seamless execution across diverse hardware. However, a subtle and often undocumented behavior can catch engineers off guard: the silent conversion of your carefully trained FP32 (float32) model to FP16 (float16) precision. This automatic optimization, while beneficial for speed and memory, can introduce unexpected numerical drift and degrade model accuracy if not anticipated and managed.

The Drive for Efficiency: Why FP16 Conversion Happens

FP16, or half-precision floating-point, uses 16 bits of memory per value compared to FP32's 32 bits. This halves memory bandwidth requirements and can dramatically accelerate computation on modern hardware like Apple's Neural Engine, NVIDIA GPUs (with Tensor Cores), and mobile NPUs. The performance gains are compelling—often 1.5x to 3x faster inference with lower power consumption.

Both ONNX Runtime and CoreML are designed to leverage these hardware capabilities aggressively. When you load a model, their graph optimizers and backend executors analyze the compute graph and available hardware. If the system detects support for efficient FP16 math, it may automatically insert conversion nodes or instruct the hardware to perform computations in FP16, even if your original model is defined in FP32. This process is frequently "silent"—it happens under the hood without explicit warnings in standard logs, presented as a beneficial optimization.

Unseen Risks: When Silent FP16 Conversion Bites

The primary risk is numerical precision loss. The reduced dynamic range and precision of FP16 can cause several issues:

• Accuracy Degradation: Models sensitive to small activation values (e.g., some NLP transformers, models using softmax with large logit ranges) may see a measurable drop in accuracy. A model scoring 95.0% in FP32 might drop to 93.5% in FP16.
• Vanishing/Exploding Activations: Values outside FP16's range (~±65,000) will clip to infinity or zero, potentially breaking model execution.
• Non-Deterministic Behavior: The cumulative effect of many low-precision operations can lead to subtle, hard-to-reproduce differences in output between runs or devices.

Furthermore, this conversion can create a debugging nightmare. The discrepancy between your training/validation environment (typically FP32) and the deployment environment (silently FP16) becomes a hidden variable, making it difficult to diagnose why a perfectly good model underperforms in production.

Taking Control: How to Manage Precision in Deployment

Being proactive is key. Here’s how to ensure precision behavior aligns with your model’s requirements:

For ONNX Runtime:
Explicitly set the session configuration options. Use GraphOptimizationLevel.ORT_DISABLE_ALL to prevent all optimizations initially, or, more precisely, control the execution provider (EP) settings. For example, with the CUDA EP, you can set enable_cuda_graph and precision flags. The most robust method is to pre-convert your model to your desired precision (using tools like onnxconverter-common) and then disable optimizations that might alter it.

For CoreML:
When converting a model to CoreML format using coremltools, you have explicit control. Use the compute_precision parameter in the conversion API (e.g., ct.transform.CoreMLComputePrecision.FLOAT32) to lock precision. For models already in CoreML format, you can inspect the .mlmodel package contents or use Netron to check layer data types, though runtime behavior is ultimately controlled by the OS and hardware.

Universal Best Practices:
1. Validate Quantitatively: Always run inference on a representative validation set through your deployment pipeline and compare outputs bit-for-bit with your FP32 reference. Measure the delta.
2. Document Your Stack: Record the exact versions of ONNX Runtime, CoreML, OS, and hardware used, as optimization behaviors can change.
3. Consider Explicit Quantization: Instead of relying on silent conversion, use formal quantization-aware training (QAT) or post-training quantization (PTQ) tools to produce a robust, explicitly FP16 or INT8 model. This gives you control and allows for calibration to mitigate accuracy loss.

Key Takeaways

• ONNX Runtime and CoreML may automatically convert FP32 models to FP16 to optimize for supported hardware, often without explicit notification.
• This conversion can improve inference speed and reduce memory usage but risks numerical instability and accuracy loss for precision-sensitive models.
• Engineers must proactively control precision settings during model conversion and session configuration, not assume FP32 preservation.
• Rigorous validation comparing deployment outputs to original model outputs is non-negotiable for production-critical systems.

In the pursuit of optimal performance, automation is a double-edged sword. The silent FP16 conversion in ONNX Runtime and CoreML is a powerful feature, but it demands respect and understanding. By moving from a passive to an active stance—explicitly defining precision requirements, validating thoroughly, and leveraging formal quantization tools—ML engineers can harness the speed of FP16 without surrendering the accuracy their models deserve. The responsibility for precision, like all aspects of deployment, ultimately rests with the practitioner.