Deployment runtimes: ExecuTorch and TensorFlow Lite Micro
This section provides information on the two on-device runtimes that DEEPCRAFT™ Model Converter supports: TensorFlow Lite Micro (TFLM) and ExecuTorch. It compares how each runtime performs, lists the features each runtime supports, and helps you select the right runtime for your model.
DEEPCRAFT™ Model Converter prepares a trained model for embedded deployment. The runtime then executes the model on the target device. You can select one of the following runtimes:
- TensorFlow Lite Micro (TFLM): The default and production-ready runtime.
- ExecuTorch: An alternative, experimental runtime for PyTorch models. To use ExecuTorch, enable the Use ExecuTorch checkbox in the GUI or add the
--executorchoption in the CLI.
The two runtimes differ in how they represent the model and in the on-device interpreter that executes it.
Summary
The table below summarizes the key differences between the two runtimes:
| Question | Answer |
|---|---|
| Which runtime should I use? | Use TFLM, which converts PyTorch models through LiteRT-Torch. TFLM is the recommended default. |
| Can I use ExecuTorch? | Yes, but ExecuTorch is experimental. Expect larger memory arenas and more failed deployments than with TFLM. |
| Which input model works? | Only a natively exported PyTorch model. Models routed through ONNX (Keras → ONNX → PyTorch) are not viable: worst results on every metric, and recurrent layers cannot be converted at all. |
| When is ExecuTorch worth it? | Very small models where runtime library size dominates, models needing better accuracy or models that fail to export through LiteRT-Torch |
| Biggest ExecuTorch win (MLPerf Tiny set) | A depthwise-separable CNN keyword spotter: −19 % NPU cycles, −2 % CPU cycles, −96 % activation buffer vs TFLite. |
| Biggest ExecuTorch loss (MLPerf Tiny set) | A residual CNN (ResNet-style) image classifier: 66 NPU operators vs 41, and roughly +543 % NPU cycles. |
| Biggest ExecuTorch win (LiteRT-Torch comparison) | A YOLOv8-nano body-pose model with a split graph: −56 % NPU cycles, −5 % CPU cycles, −4 % arena. |
| Biggest ExecuTorch loss (LiteRT-Torch comparison) | A small dense model: +432 % arena, +366 % NPU cycles, +271 % CPU cycles. |
| Runtime library size | ExecuTorch can be roughly 3× smaller (≈91 kB vs ≈267 kB), because only the kernels the model actually uses are linked. |
| Deployment success rate | 83.33% native .pt2 models deployed through TFLM vs 58.33% through ExecuTorch. |
Runtime workflows
TensorFlow Lite Micro (default)
DEEPCRAFT™ Model Converter converts the source model, such as TensorFlow Keras (.h5, .keras), TensorFlow Lite (.tflite), or PyTorch (.pt2), into a quantized TensorFlow Lite model. The TFLM runtime then executes the model on the target device. For PyTorch models, Model Converter first converts the model to TensorFlow Lite using LiteRT-Torch, then optimizes and quantizes it. This workflow supports Int8x8 and Int16x8 quantization, interpreterless mode, and NPU acceleration on Ethos-U55 and NNLite.
ExecuTorch (alternative)
DEEPCRAFT™ Model Converter exports the PyTorch (.pt2) model to an ExecuTorch program (.pte), or uses a pre-exported .pte model directly. The ExecuTorch runtime executes the program on the target device, along with generated code that registers the operator kernels. Because the .pte program can include input and output quantization, the model inputs and outputs use their native data types. This workflow supports Int8x8 quantization and NPU acceleration on Ethos-U55 only.
TFLM: .h5 / .keras / .tflite / .pt2 ──► .tflite ──► TFLite Micro
ExecuTorch: .pt2 / .pte ──► .pte (+ kernel registration) ──► ExecuTorch runtimeCurrent status of the runtimes
For most PyTorch models, the PyTorch → TensorFlow Lite → TFLM workflow currently produces smaller, faster, and better-optimized deployments than ExecuTorch. TFLM also supports the full set of Model Converter features, including all quantization types, interpreterless mode, NPU acceleration, and reporting. ExecuTorch works, but it does not yet match TFLM on memory footprint or performance, and it does not support interpreterless mode or Int16x8 quantization.
The table below compares the features of the two runtimes:
| Feature | TFLM (default) | ExecuTorch (experimental) |
|---|---|---|
| Source models | TensorFlow Keras, TensorFlow Lite, PyTorch | PyTorch |
| On-device format | .tflite | .pte program |
| Maturity | Production | Experimental |
| Memory footprint and speed | Best overall | Usually larger and slower, occasionally better |
| Runtime library size | Fixed, ≈267 kB | Links only the required kernels, ≈91–120 kB |
| Quantization | Int8x8, Int16x8 | Int8x8 |
| NPU acceleration | Ethos-U55, NNLite | Ethos-U55 only |
| Interpreterless mode | Supported | Not supported |
| Operators that run on the CPU | Quantized kernels | Float32 reference kernels (slower) |
The Ethos-U55 NPU executes only the operators listed in Arm’s Ethos-U55 supported operator list . The Cortex-M55 CPU executes all other operators.
Benchmark results
The following sections provide detailed benchmark data for advanced users. The tables above summarize the key findings.
MLPerf Tiny benchmark
The benchmark ran on the PSOC™ Edge AI Kit with the PSE84 device (Cortex-M55 and Ethos-U55 with 128 MACs at 400 MHz). It used the MLPerf Tiny reference networks, re-implemented natively in PyTorch with weights transferred from the Keras reference models.
| Architecture | Task | ExecuTorch compared to TFLM |
|---|---|---|
| Fully connected autoencoder (640→…8→…→640) | Anomaly detection (regression) | Comparable |
| Depthwise-separable CNN (DS-CNN) | 12-class keyword spotting | −19% NPU cycles, −2% CPU cycles, −96% activation buffer |
| Residual CNN (ResNet-style) | Image classification (CIFAR-10) | 66 NPU operators compared to 41 (+61%), about +543% NPU cycles |
| MobileNetV1-style CNN (13 depthwise-separable blocks) | Visual wake words | Comparable |
| Small dense network (DNN) | 4-class wake word detection | Comparable |
Across the wider internal benchmark set of 15 models, ExecuTorch showed the following results compared to TFLM:
- Model file size: Comparable at best and occasionally much larger, up to +240% for one audio model.
- Tensor arena: Occasionally slightly smaller, but several models need 2–5× the TFLM arena. On PSE84, this can push the inference working set beyond the available SOCMEM.
- NPU cycles: Typically 5–15% higher and occasionally 50% or more higher. Models with layout transposes that the compiler cannot remove perform considerably worse.
- CPU cycles: Often 5–10× higher, because operators that the NPU does not execute run in Float32 instead of using quantized kernels. Arm is developing quantized CPU kernels for ExecuTorch.
- Accuracy: Higher than TFLM on every measured model, because ExecuTorch runs non-delegated operators in Float32 and avoids the quantization error that TFLM introduces.
- Runtime footprint: ExecuTorch links only the registered kernels, which requires ≈91 kB compared to ≈267 kB for the TFLM interpreter. For a 100 kB model, this reduces the total image size considerably.
ExecuTorch uses more memory and cycles for the following reasons:
- Arm designed the Ethos-U55 compiler around TensorFlow Lite and added ExecuTorch support later, using a separate high-level optimization pipeline. As a result, the same compiler version can generate different, and usually less efficient, NPU command streams for the two runtimes.
- PyTorch uses the NCHW layout, while Ethos-U uses the NHWC layout natively. The compiler inserts extra transposes to convert between the layouts, which increases the buffer size the NPU requires.
Native PyTorch models deployed with both runtimes
This benchmark deployed natively exported PyTorch models on PSE84 (Cortex-M55 and Ethos-U55) twice: once through LiteRT-Torch and TFLM, and once through ExecuTorch. Both deployments used the same NPU settings: Memory Mode set to Shared SRAM, Optimize set to Performance, and Tensor Allocator set to HillClimb.
The table below shows ExecuTorch results relative to the TFLM baseline. A positive value means that ExecuTorch requires more resources. Unless noted otherwise, the cycle counts are measured with the weights stored in SOCMEM.
| Model (architecture) | Weights | Arena | NPU cycles | CPU cycles | Accuracy (MAE) |
|---|---|---|---|---|---|
| YOLOv5-nano, 192×192 | +1% | −1% | +5% | ~+650% | Worse (≈14.6 compared to 2.7) |
| YOLO11-nano, 192×192 | +1% | +19% | +20% | ~+1000% | Better (≈0.67 compared to 2.44) |
| YOLO26-nano, 224×224, split head | +1% | +216% | +29% | ~+1900% | Better (≈1.58 compared to 3.38) |
| YOLOv8-nano body pose, 224×224 (Ultralytics export) | +1% | +17% | +12% | ~+650% | Better (≈0.48 compared to 2.40) |
| YOLOv8-nano body pose, 224×224 (split graph, weights in flash) | +1% | −4% | −56% | −5% | Worse (≈1.72 compared to 1.11) |
| Rock-paper-scissors YOLOv5 | +1% | +156% | +15% | ~+1900% | Much better (below 0.4 compared to 8–52) |
| Small dense model | +61% | +432% | +366% | +271% | Equal (both ≈0) |
The results show the following patterns:
- Model size: For convolutional networks, ExecuTorch and TFLM produce models of nearly the same size (about +1%). Small non-convolutional models are the exception, where the ExecuTorch weight representation can be several times larger.
- Tensor arena: The arena size is the main risk. It ranges from slightly smaller than TFLM to about 3× larger. ExecuTorch splits the arena into two pools: planned memory and Ethos-U scratch memory. Two audio models failed to run because the Ethos-U delegate requested a scratch buffer of about 7 MB, while SOCMEM provides only 5 MB.
- NPU cycles: Usually 5–30% higher. However, one split-graph pose model ran about 2× faster on the NPU with ExecuTorch.
- CPU cycles: Consistently several times higher, commonly 4–10× and occasionally about 20×, because non-delegated operators run as Float32 reference kernels. This is the main cause of higher end-to-end latency.
- Accuracy: Usually higher with ExecuTorch, sometimes by an order of magnitude, because the non-delegated operators remain in Float32 when you apply post-training quantization (PTQ).
- Runtime library size: Consistently smaller with ExecuTorch, about 95–120 kB of linked kernels compared to a fixed ≈268 kB for the TFLM interpreter.
- Deployment success rate: Higher with TFLM. 83.33% of PyTorch (.pt2) models deployed successfully through TFLM, compared to 58.33% through ExecuTorch.
Models that failed to deploy
The table below lists the models that failed to deploy with one or both runtimes:
| Model (architecture) | TFLM | ExecuTorch | Cause |
|---|---|---|---|
| Two audio magnitude-spectrum models | Deployed | Failed | The Ethos-U delegate requests a scratch buffer of about 6.9 MB, but SOCMEM provides 5 MB and the Cortex-M55 heap region provides about 4.4 MB. The weights are also about 5–6× larger than with TFLM. |
| U-Net speech enhancement | Deployed (weights in flash, arena of about 4.5 MB) | Failed | Kernel generation rejects a bias keyword argument that resolves to a graph node instead of a constant. |
| TTS vocoder (iSTFTNet-style) | Deployed | Failed | Model conversion stops during export. |
| DeepFilterNet2 speech enhancement | Failed | Failed | LiteRT-Torch conversion fails, and the ExecuTorch workflow fails in the Arm quantizer. |
| MobileCLIP-S0 image encoder | Failed (inference stops while waiting for the NPU) | Builds, with about +71% arena | The model is larger than 11 MB and does not fit in SOCMEM. |
Note the following when troubleshooting failed deployments:
- Both runtimes share several failure causes, such as unsupported input data types, operators that the quantizer rejects, or exported graphs that include pre-processing or post-processing code. These are model preparation issues, not runtime issues.
- A failure can depend on how you export the model. One YOLO detector failed with both runtimes when exported directly from a live model, but deployed successfully after re-export with constant-folded shape operations. A body-pose model required the Ultralytics export workflow, because the generic conversion generated an operator that the NPU compiler does not accept.
Selecting a runtime
The table below recommends a runtime for common scenarios:
| Scenario | Recommendation |
|---|---|
| New deployment with no special constraints | Use LiteRT-Torch and TFLM. TFLM has a higher deployment success rate and more predictable memory and arena usage. |
| Runtime library size is the main constraint | Try ExecuTorch. The ExecuTorch runtime can be several times smaller. |
| The model loses accuracy after quantization | Try ExecuTorch. Non-delegated operators run in Float32, which can improve the MAE. |
| LiteRT-Torch export fails for your model | Try ExecuTorch. |
| The model uses LSTM or other recurrent layers | Use TFLM. The ONNX-based conversion workflow does not support recurrent layers. |
Preparing a PyTorch model for ExecuTorch
Do not convert your model through ONNX. Re-implement the model natively in PyTorch and transfer the weights instead.
Staged conversion (Keras → ONNX → onnx2torch → torch.export → .pt2) produced the worst results on every metric. For the keyword-spotting model, staged conversion increased the graph to 99 operators, compared to 34 for the native PyTorch equivalent. It also changed the parameter count (22,610 compared to 24,917) because of differences in BatchNorm fusion, and it failed on LSTM layers. Native re-implementation with transferred weights reproduced the Keras reference models within a maximum absolute difference of 1×10⁻⁶, with a maximum relative error below 0.02% on all models except the autoencoder.
Model authoring guidelines
The following guidelines measurably improve ExecuTorch deployments:
| Guideline | Reason |
|---|---|
Use F.adaptive_avg_pool2d(x, (1, 1)) instead of a fixed-kernel F.avg_pool2d. | The operator maps to aten.adaptive_avg_pool2d in the compiled graph, which is cleaner and more portable. The results do not change numerically. |
Use non-in-place ReLU (relu or nn.ReLU(inplace=False)). | In-place operators on residual-add outputs modify aliased storage. ExecuTorch ahead-of-time tracing is sensitive to aliasing and can build an incorrect graph. The results do not change numerically. |
Set Tensor Format to Channels Last (CLI: --tensor-format channels_last) when your dataset uses NHWC and the model uses NCHW. | The setting describes the layout of the calibration and validation data, not the model. Do not wrap the model in an input-transpose layer, because the layer adds unnecessary operators. |
Export YOLO models with the Ultralytics Export API instead of manually running torch.export on a live model. | Manual export includes pre-processing and post-processing in the graph, which breaks both ExecuTorch and LiteRT-Torch. Provide the exported .pt2 model to Model Converter, which applies the correct NPU configuration for the target. |
| Check the output graph for non-delegated operators. | ExecuTorch and TensorFlow Lite differ in operator coverage, decomposition, and fusion. A model that TFLM fully delegates to the NPU can still have CPU operators with ExecuTorch, and those operators run as Float32 reference kernels. Refer to the Ethos-U55 supported operator list . |
To export models reliably, the reference implementation tries the following export modes in order until one succeeds: strict=False with runtime_asserts, strict=False, draft_export, and strict=True.
Recommended NPU (Ethos-U) settings
The benchmark compared five NPU configurations on the autoencoder model. The performance-oriented, SRAM-only configuration produced the best results. You can configure these settings under Advanced Options in the GUI:
| GUI option | Recommended value | CLI option |
|---|---|---|
| Optimize | Performance | --npu-opt-strategy Performance |
| Memory Mode | SRAM Only | --npu-mem-mode SramOnly |
| Tensor Allocator | HillClimb (default) | --npu-tensor-alloc HillClimb |
| Arena Cache Strategy and Arena Cache Size | Keep the default values | --npu-arena-strategy, --npu-arena-cache |
Changing the arena cache size or setting the tensor allocator to Greedy had no measurable effect.
Residual networks and layout transposes
When you export a residual CNN in the default NCHW layout, the exporter inserts transposes at the model input and around each residual addition, where the skip path and the main convolution path use different memory layouts. The ExecuTorch model contained 66 NPU operators, compared to 41 for the TFLM equivalent (+61%), and used about +543% more NPU cycles.
Exporting in the channels-last (NHWC) layout did not reduce the operator count, which remained at 66. The downstream compiler had already removed the transposes at the model input. The remaining transposes are structural: the layout mismatch at each residual addition creates them, and changing the export format cannot remove them.
Approaches that are not recommended
The table below lists approaches that the benchmarks showed to be ineffective:
| Approach | Reason |
|---|---|
| Converting models to ExecuTorch through ONNX | Produces the worst results on all metrics, increases graph size because of the ONNX intermediate representation, and does not support LSTM layers. |
| Adding an NHWC input-wrapper layer to NCHW models | Adds unnecessary operators. Use the Tensor Format option instead. |
| Exporting residual networks in channels-last layout to remove transposes | The skip-path transposes remain, and the operator count stays at 66. |
| Setting Tensor Allocator to Greedy | Shows no meaningful difference compared to HillClimb. |
| Setting a custom Arena Cache Size | Shows no measurable effect. |
Future of ExecuTorch
ExecuTorch is the native edge runtime for PyTorch, with significant investment from Meta and Arm. As the toolchain matures, ExecuTorch is expected to close the gap with TFLM and outperform it in some areas. Planned improvements include a direct PyTorch-to-device workflow without TensorFlow Lite conversion, first-class PyTorch operator coverage, quantized CPU fallback kernels (in development at Arm), and tighter integration with the Arm backend and NPU. DEEPCRAFT™ Model Converter offers ExecuTorch now, so you can evaluate the PyTorch-native workflow early and adopt it as it improves.
Currently, TFLM is likely to produce a smaller and faster deployment, especially when the Ethos-U55 does not support some operators and they fall back to the CPU.