Introduction to Dataset Node

DEEPCRAFT™ Studio is moving toward an end-to-end edge machine learning workflow in Graph UX. As part of this, Graph UX now includes a new Dataset node. The Dataset node is a self-contained place to collect, import, label, and manage all your data inside the graph, so your data is no longer scattered across folders and files.

Key terms

• Session: A single, complete recording in DEEPCRAFT™ Studio. A session captures everything recorded during one recording period, such as sensor signals, audio, metadata, and labels, as a set of time-aligned tracks. Each time you collect real-time data or import your own data, Studio saves it as a session.
• Track: One stream of time-based data within a session. A session can have multiple tracks, and they stay aligned in time so you can view and label the recording.
• Data track: A track that stores a sensor signal.
• Label track: A track that stores the classes you assign to the data in a data track.
• Timeline: The view that shows a session’s tracks along a horizontal time axis, from the start of the recording to the end. Use the timeline to navigate the recording, inspect data at specific points in time, and create or edit labels.

How the Dataset node differs from the existing workflow

In the existing workflow, you collect and label data by connecting a sensor node to a Track node and defining classes in a Predefined Labels node. You then save the sessions and add the collected data to a Studio project to train the model.

With the Dataset node, all of this happens inside one node:

• When you connect a sensor node to the Dataset node, the data tracks are created automatically, so you can start collecting data right away.
• You create label tracks and define their classes directly in the Dataset node.
• You can import existing data into the same Dataset node.
• You manage sessions and tracks inside the node.
• When your data is ready, you send it straight to a training project in DEEPCRAFT™ Studio. You do not need to move the data to the workspace or add the data to a project manually.

Dataset node modes

A Dataset node works in one of two modes, depending on how you connect the dataset node in the graph:

• Data Collection mode: In this mode, you use the Dataset node to collect, import, and label data. To collect real-time data, connect the Dataset node to a sensor node. To import your own data, use any Dataset node, whether it is connected to a sensor node or not.

• Data Processing mode: The Dataset node is connected to another Dataset node, with processing nodes in between. Use this mode to process the data in the source Dataset node and save the result in the target Dataset node. This mode is used to process the data in a batch.

In Data Processing mode, you cannot collect or import data into the target Dataset node. You can only sync data from the source Dataset node to the target Dataset node.

Batch processing with Dataset nodes

Besides collecting and labeling data, a Dataset node can process data in batches. You can augment, clean, or otherwise preprocess all the data in a dataset in one operation.

To process data in a batch:

1. Connect the augmentation or data processing nodes you need to the output of the Dataset node that holds your collected and labeled data.
2. Connect a second Dataset node to the output of the last processing node.
3. Sync the two Dataset nodes.

When you sync, the data from the first Dataset node runs through the processing nodes, and the result is saved in the second Dataset node. You can then send the processed data straight to a training project in DEEPCRAFT™ Studio to train your model.

Improved timeline performance

The Dataset node’s timeline performs better with large volumes of data. It no longer renders every data point, so zooming, scrolling, and navigating are smoother. You can also record data from high-bandwidth sensors, such as radar, without slowdowns or crashes.

Getting started with the Dataset node

The Dataset node handles the first part of the machine learning workflow: getting your data ready for training. The rest of the workflow happens in a DEEPCRAFT™ Studio training project.

In the Dataset node:

1. Collect real-time data from a sensor, or import existing data into the Dataset node. For more information, refer to Data Collection.
2. Label the data. For more information, refer to Data Labeling.
3. Export the data to a training project. For more information, refer to Exporting data to a training project from Dataset Node

After you export the data, follow the standard machine learning workflow in DEEPCRAFT™ Studio:

1. Redistribute the data across the training, validation, and test sets. For more information, refer to Data Distribution.
2. Preprocess the data, if needed. For more information, refer to Preprocessing.
3. Train the model. For more information, refer to Model Training.
4. Generate code from the trained model. For more information, refer to Code Generation.
5. Deploy the model to your edge device. For more information, refer to Deployment.

Managing tracks in the Dataset node

Automatic track creation

When you connect a node to a Dataset node, DEEPCRAFT™ Studio automatically adds a track for that node to the Tracks tab. For example, if you connect a microphone node, a microphone track is created. Each node you connect gets its own track, so a Dataset node with several connected nodes has one track per node.

Removing a connection

Removing the wire between a node and the Dataset node in the graph does not delete the node’s track. The track and all data recorded through it stay in the Dataset node.

Deleting a track

When you delete a track from the Tracks tab, all data recorded in that track is permanently removed from every session that contains it. For example, if you delete the Local Microphone track, all data recorded by the local microphone is deleted. What happens to each session depends on how many tracks it has:

• Session with a single track: The entire session is deleted, because the deleted track was its only data.
• Session with multiple tracks: Only the data for the deleted track is removed. The session is kept, and the data from its other tracks stays as it is.

Managing Dataset nodes with .dcdb files

A DEEPCRAFT Database file (.dcdb) is the storage behind a Dataset node. It contains all the data in the node, including sessions and tracks. Every Dataset node has its own .dcdb file.

How .dcdb files are created

When you create a Dataset node on the graph, DEEPCRAFT™ Studio creates a matching .dcdb file in the Dataset folder of your project in the workspace. The node and the file are linked, so if you delete the .dcdb file, the Dataset node is also deleted.

Deleting a Dataset node

When you delete a Dataset node, a prompt asks whether you want to delete or keep the data in the node:

• Yes, delete the files: The Dataset node and all the data, including recorded sessions and tracks, are permanently deleted.
• No, keep the files: The Dataset node is removed from the graph, but the data, including recorded sessions and tracks, stays in the .dcdb file in the Dataset folder of your workspace.

Restoring a Dataset node

To use the dataset again, drag the .dcdb file from the Dataset folder in the workspace onto the graph. This creates a new Dataset node that contains all the data stored in the file.