Deployment API

This section provides information on the two deployment APIs available in DEEPCRAFT™ Model Converter: Imagimob API and ml-middleware API. You can choose one of the two APIs to deploy your model on the target device.

Choosing the Firmware API for your application

When generating code in DEEPCRAFT™ Model Converter, you can select which API your firmware will use to run the model. Both the Imagimob API and the ml-middleware API convert and quantize the model identically, and both run on Infineon’s ModusToolbox™ ML middleware at runtime. The difference is how much integration code the Mode Converter generates for your application.

• ml-middleware: When this option is selected, the Model Converter generates only the model artifacts, so your application calls the Infineon’s mtb_ml_* API directly. You perform the model initialization, arena setup, and inference in your firmware. By default, this option is selected as the target API when you generate code.

• Imagimob API: When this option is selected, the Model Converter generates the model artifacts in addition to a small, self-contained model.c and model.h. These files expose the IMAI_* functions and wrap initialization, arena setup, quantization, and inference behind IMAI_init() and IMAI_compute().

Both options generate a code_generation_report.md. Depending on the API you select, the output includes either the raw middleware artifacts or the IMAI_* model files. You can select the API in the GUI, in the project file, or with the corresponding CLI flag.

The deployment APIs support the following quantization on inputs and outputs - float, 8-bit and 16-bit integer To use the generated artifacts, the following COMPONENTS and DEFINES are required:

The solution implements its own version of the CMSIS_NN library, therefore do NOT add CMSIS_NN in the component list in the Makefile, only the header files, as the example below: INCLUDES+=$(SEARCH_cmsis)/COMPONENT_CMSIS_NN/Include

Ml-middleware API

The Model Converter generates model artifacts that the ModusToolbox™ ML middleware uses directly. The middleware abstracts the underlying TensorFlow Lite Micro runtime and provides a C API for model setup, input handling, and inference as shown below:

mtb_ml_model_init() : to initialize the model (call once)
mtb_ml_model_get_output_tensor() : to obtain the output buffer pointer and its size (call once on every output)
mtb_ml_model_inputs() : to feed data for each input (call on every sample)
mtb_ml_model_invoke() : to run the inference engine (call on every sample)

Include the generated model files

Include the generated model files using the helper macros. In the example below, test_model is the output prefix generated by the Model Converter. Include the ml-middleware header first, because it defines both the API and the required macros.

#include "mtb_ml.h"                                     // ml-middleware API + macros
#include MTB_ML_INCLUDE_MODEL_FILE(test_model)         // model weights/params
#include MTB_ML_INCLUDE_MODEL_X_DATA_FILE(test_model)   // regression input (optional)
#include MTB_ML_INCLUDE_MODEL_Y_DATA_FILE(test_model)   // regression output (optional)
 
// accessor macros
MTB_ML_MODEL_ARENA_SIZE(test_model)  // TFLM required arena size
MTB_ML_MODEL_NAME_STR(test_model)      // string for the model name
MTB_ML_MODEL_BIN_DATA(test_model)        // populates the model binary structure
MTB_ML_MODEL_X_DATA_BIN (test_model) // regression data input array
MTB_ML_MODEL_Y_DATA_BIN (test model)         // regression data output array

Initialize the model

To initialize the model, call mtb_ml_model_init() exactly once before running inference. You can let the middleware allocate memory internally, or you can provide your own tensor arena.

The following table shows the steps for the two initialization methods:

Method 1: Using internal memory allocation

mtb_ml_model_t *model_object;
/* NN model data */
mtb_ml_model_bin_t model_bin = {MTB_ML_MODEL_BIN_DATA(test_model)};
/* Initialize the model */
mtb_ml_model_init(&model_bin, NULL, &model_object);

Method 2: Using external memory allocation

 
mtb_ml_model_t *model_object;
uint8_t tensor_arena[MTB_ML_MODEL_ARENA_SIZE(test_model)];
mtb_ml_model_buffer_t mem_buf = {tensor_arena,
MTB_ML_MODEL_ARENA_SIZE(test_model)};
/* NN model data */
mtb_ml_model_bin_t model_bin = {MTB_ML_MODEL_BIN_DATA(test_model)};
/* Initialize the model */
mtb_ml_model_init(&model_bin, &mem_buf, &model_object);

Choose this option when you want full control over model integration inside a ModusToolbox™ application. For a complete configuration example, see the Readme and makefiles in the deployment Code Example.

Using the TFLM API Directly

The ml-middleware option generates the model artifacts directly, you are not required to use the mtb_ml_* abstraction. You can integrate the generated files with the underlying runtime yourself by using the native TensorFlow Lite Micro (TFLM) C++ interpreter , depending on the target selected during model generation. This approach is useful if your application already has an inference stack, requires runtime behavior that the middleware does not expose, or needs to bypass the middleware layer. However, on the NPU targets, you must still call mtb_ml_init() before inference to initialize the NPU drivers. This requirement applies even when you use the TFLM runtime directly.

Running Inference

To run inference with the ml-middleware API, set each input tensor with mtb_ml_model_inputs(), then run inference withmtb_ml_model_invoke(). Read each output with mtb_ml_model_get_output_tensor(). The output layer is capped at 64 nodes.

You can feed input data to the model in either of two formats:

• Use data that matches the model’s native tensor type.
• Use floating-point data and let the middleware quantize the data before inference.

For floating-point inputs, call mtb_ml_utils_model_quantize_tensor() to quantize the data using the model’s scale and zero-point. After inference, call mtb_ml_utils_model_dequantize_tensor() to convert native output tensors back to floating-point values.

// Inputs — already native:        mtb_ml_model_inputs(model_object, native_input[i], i);
// Inputs — float, quantize first:
for (int i = 0; i < input_count; i++) {
    mtb_ml_utils_model_quantize_tensor(model_object, i, src_f32[i], native_input[i], count_in[i]);
    mtb_ml_model_inputs(model_object, native_input[i], i);
}
 
mtb_ml_model_invoke(model_object);
 
for (int i = 0; i < output_count; i++) {
    MTB_ML_DATA_T *out; int size;
    mtb_ml_model_get_output_tensor(model_object, &out, &size, i);
    mtb_ml_utils_model_dequantize_tensor(model_object, i, dst_f32[i], size); // native → float
}

Imagimob API: Auto-Generated Wrapper

The Imagimob API option generates model.c and model.h, which expose a compact, portable C API for running the model. The generated functions wrap the middleware used by the ml-middleware option, while handling model initialization, tensor arena setup, and input quantization.

Core API

The generated API includes three main functions:

int  IMAI_init(void);                           // Initialize the model
void IMAI_compute(in0, in1, ..., out0, ...);   // Run inference
void IMAI_finalize(void);                       // Free resources

IMAI_compute() runs inference in a single call. Its arguments list all model inputs, followed by all model outputs.

The API returns the following status codes: IMAI_RET_SUCCESS (0), IMAI_RET_NODATA (-1), and IMAI_RET_ERROR (-2). The default function prefix is IMAI_, but you can configure prefix during code generation.

Choose the Imagimob API option when you want minimal, drop-in integration. The generated wrapper still links against the ModusToolbox™ ML middleware, so your project must include the required ModusToolbox™ COMPONENTS, such as ML_TFLM, the selected CPU or NPU target, and optionally CMSIS_DSP. The generated code already contains the selected quantization type, so you do not need to add separate precision components.

Auto-quantization is optional. By default, IMAI_compute() accepts floating-point input and quantizes it before inference. If your application already provides quantized input data, you can disable auto-quantization and use the generated IMAI_quantize() and IMAI_dequantize() helpers manually. These helpers use each tensor’s scale and zero-point. See the multi-IO example below.

Arena and Weights Placement

The Imagimob API lets you choose where the tensor arena and model weights are stored. IMAI_init() applies the selected configuration automatically.

• Arena allocation: Use Static allocation, the default, to place a fixed tensor arena buffer in the binary. Use Dynamic (Heap) allocation to allocate the arena with malloc() at runtime, so it consumes RAM only while the model is loaded.

• Weights placement: Keep weights in flash for read-only access without a RAM copy, or choose Dynamic placement to store the weights in flash and copy them to RAM automatically during IMAI_init(). Dynamic placement can improve performance when running from RAM is faster than execute-in-place from flash.

With the ml-middleware option, your application manages the tensor arena and weights placement directly.

Multiple inputs and outputs models

For multi-input and multi-output models, IMAI_compute() lists all input tensors first, followed by all output tensors.

IMAI_compute(in0, in1, ..., out0, out1, ...);

For a known model, call the function directly with the explicit arguments. For example, a model with two inputs and three outputs would use:

IMAI_compute(in0, in1, out0, out1, out2);   // IMAI_COMPUTE_INPUTS=2, IMAI_COMPUTE_OUTPUTS=3

The generated header also advertises the counts, so a generic loop can fill inputs and read outputs and dispatch through IMAI_COMPUTE_PTR:

void *args[IMAI_COMPUTE_INPUTS + IMAI_COMPUTE_OUTPUTS];           // inputs first, then outputs
for (int i = 0; i < IMAI_COMPUTE_INPUTS; i++)  args[i] = input_ptr[i];
for (int i = 0; i < IMAI_COMPUTE_OUTPUTS; i++) args[IMAI_COMPUTE_INPUTS + i] = output_ptr[i];
IMAI_COMPUTE_PTR(args);                                            // calls IMAI_compute(in0, …, out0, …)

The model metadata accessors, IMAI_INPUT_META(i) and IMAI_OUTPUT_META(i), provide each tensor’s size, scale, and offset. Use these values to perform manual quantization for each tensor individually.

// Inputs: float → native, quantized per tensor before compute
for (int i = 0; i < IMAI_COMPUTE_INPUTS; i++)
    IMAI_quantize(in_f32[i], in_q[i], IMAI_INPUT_META(i)->count,
                  IMAI_INPUT_META(i)->scale, IMAI_INPUT_META(i)->offset);
 
IMAI_compute(in_q[0], in_q[1], out_q[0], out_q[1], out_q[2]);
 
// Outputs: native → float, dequantized per tensor
for (int i = 0; i < IMAI_COMPUTE_OUTPUTS; i++)
    IMAI_dequantize(out_q[i], out_f32[i], IMAI_OUTPUT_META(i)->count,
                    IMAI_OUTPUT_META(i)->scale, IMAI_OUTPUT_META(i)->offset);

Model Placement

When using the ml-middleware API or Imagimob API with weights placement set to Makefile Defined (default), you can specify the memory section used to store the model binary. Device-specific recommendations are listed below; any custom memory section may also be used.

When using PSOC™ 6

To store the model in internal flash or external memory, add one of the following defines to the Makefile:

Refer to the https://github.com/Infineon/mtb-example-PSOC6-qspi-xip  code example to properly setup the XIP mode using QSPI.

When using PSOC™ Edge

To store the model in internal flash or external memory, add one of the following defines to the Makefile:

Additional deployment instructions for PSOC™ Edge U55 NPU targets

When using Ethos-U55 with CM55, the underneath TFLM implementation, clear/invalidate the CM55 D-Cache to avoid any discrepancy in the memory (as the latest data might be only available in the cache or actual memory). This is done for every TFLM operator. However, if all operators are handled by the Ethos-U55, there is no need to clear/invalidate the cache on every operator, only in the input and output layer. You can add the following macro in the CM55 Makefile DEFINES list to disable these cache operations:

DEFINES+= MTB_ML_ETHOSU_CACHE_MGMT_TYPE=MTB_ML_ETHOSU_CACHE_MGMT_OUTER_LAYERS

If you see any changes in the inferencing results, that means you should NOT enable this cache optimization, as the CM55 and U55 are accessing the same internal buffers.

You can also call mtb_ml_set_cache_mgmt_type() function to change the cache management type to one of the below options dynamically.

• MTB_ML_ETHOSU_CACHE_MGMT_CONDITIONAL: this mode clears or invalidates the entire cache based on an internal state. It can reduce the total number of CPU cycles, but it might cause some undesired behavior in the application.
• MTB_ML_ETHOSU_CACHE_MGMT_ALL_LAYERS: this mode clears and invalidates the cache by address for each layer using cache-API calls within the driver (default option).
• MTB_ML_ETHOSU_CACHE_MGMT_OUTER_LAYERS: this mode clears the input layer before executing the inference and invalidates the output layer after executing inference. Should be only used if all operators are supported by Ethos-U55. It provides always the best performance.

The mtb_ml_set_cache_mgmt_type() function should be used when you have multiple models to run sequentially in your application, and they require different cache management types.