Understanding Real-Time Data Processing with ROS2

The Robot Operating System 2 (ROS2) has become the standard framework for building production-grade robotic systems that require deterministic, low-latency communication. Real-time data processing is essential in applications such as autonomous driving, industrial manipulation, and aerial drones, where missing a deadline can cause system failure. ROS2 addresses these challenges through its modular publisher-subscriber architecture, backed by the Data Distribution Service (DDS) middleware. Unlike ROS1, ROS2 was designed from the ground up to support real-time constraints, offering configurable quality-of-service (QoS) policies, multiple execution models, and compatibility with real-time operating systems. This guide explains how to leverage ROS2 for real-time robot data processing, from architectural foundations to advanced optimization techniques.

Core Architecture: Nodes, Topics, Services, and Actions

ROS2’s communication paradigm relies on four key abstractions:

  • Nodes are independent executables that perform computation. In real-time systems, each node typically runs in its own process, providing memory isolation and allowing the system designer to assign CPU affinities and scheduling policies per node.
  • Topics enable asynchronous publish-subscribe messaging. Data producers publish to a named topic, and consumers subscribe to receive messages. The DDS middleware handles discovery, serialization, and transport, enabling nodes to be added or removed at runtime without reconfiguration.
  • Services provide synchronous request-reply communication. While not inherently real-time, services are useful for configuration changes or parameter updates that do not require hard deadlines.
  • Actions extend services for long-running goals with periodic feedback, suitable for tasks like navigation waypoint following where intermediate status is needed.

For real-time applications, the topic-based publish-subscribe model is the most common choice because it supports decoupled communication and can be tuned for deterministic delivery via DDS quality-of-service settings.

Setting Up ROS2 for Real-Time Data Processing

Installation and DDS Selection

Start by installing a recent ROS2 distribution such as Humble Hawksbill, Iron Irwini, or Rolling Ridley on an operating system that supports real-time scheduling—Linux with a real-time kernel patch, QNX, or VxWorks. The default DDS implementation (Fast DDS) works well for many systems, but for strict real-time requirements consider alternatives:

  • Cyclone DDS – Known for low latency and high scalability; used in production autonomous vehicle systems.
  • RTI Connext Pro – A commercial DDS offering advanced real-time scheduling, security features, and zero-copy shared memory.
  • Fast DDS – The default option; recent versions include a shared memory transport with low jitter.

After installation, configure the DDS domain and QoS defaults to match your application’s timing requirements. Set the ROS_DOMAIN_ID environment variable to isolate your system from other ROS2 networks. Adjust the DDS configuration file (e.g., FASTRTPS_DEFAULT_PROFILES_FILE for Fast DDS or CYCLONEDDS_URI for Cyclone DDS) to enable shared memory transport and set thread priorities.

Real-Time Kernel and Process Scheduling

Achieving deterministic timing requires running your robot’s control loop with elevated priority. On Linux:

  1. Install a real-time kernel – for Ubuntu, use sudo apt install linux-image-rt-amd64. Alternatively, apply the PREEMPT_RT patch to your kernel.
  2. Edit /etc/security/limits.conf to allow your user to set real-time priorities, for example: youruser - rtprio 99.
  3. Assign real-time scheduling policies to your ROS2 nodes using chrt or the sched_setscheduler system call. Typically, SCHED_FIFO with a priority of 80–95 works well for control loops.

For advanced isolation, use the kernel boot parameter isolcpus to reserve CPU cores for real-time tasks, then pin each node’s threads to dedicated cores using taskset or pthread_setaffinity_np. This prevents interference from non-real-time processes and interrupts.

Developing Real-Time Data Nodes

Node Design Principles

Writing real-time nodes in ROS2 requires careful attention to resource usage inside callbacks:

  • Avoid dynamic memory allocation inside callbacks. Use pre-allocated buffers and avoid new, malloc, or STL container operations like push_back that may cause heap fragmentation. If dynamic allocation is unavoidable, perform it during node initialization, not in the hot path.
  • Keep callbacks short and deterministic. Complex processing should be offloaded to a separate thread or executor, with callbacks relegated to parsing data and publishing to new topics.
  • Use the appropriate executor. The rclcpp::executors::StaticSingleThreadedExecutor pre-allocates memory for all timers and subscriptions, providing predictable callback scheduling. For multi-threaded nodes, apply rclcpp::executors::MultiThreadedExecutor with a fixed number of threads. Avoid the default SingleThreadedExecutor for time-critical nodes because it does not guarantee deterministic ordering.

Additionally, consider using the ROS2 real-time supported executor (available in Iron and later) that respects thread priorities and avoids priority inversion during callback execution.

Implementing Quality of Service Policies

ROS2 QoS policies are the primary tool for enforcing real-time behavior. Key policies for real-time data processing include:

  • Reliability: Set to RELIABLE for guaranteed delivery of critical commands; use BEST_EFFORT for sensor data where occasional drops are tolerable but minimal latency is essential.
  • Durability: VOLATILE for transient sensor data; TRANSIENT_LOCAL for persistent state information like map data.
  • Deadline: Enforces that a message is received within a specified duration. If the publisher fails to meet the deadline, the middleware alerts subscribers. This is vital for control loops that require data at a fixed rate (e.g., every 10 ms).
  • Latency Budget: Tells the middleware the acceptable end-to-end delay. DDS implementations optimize transport accordingly—for example, by using shared memory instead of network if the budget is very tight.
  • Lifespan: Automatically discards data older than a threshold, preventing stale information from being processed.
  • History: Set to KEEP_LAST with depth 1 or 2 to minimize memory usage and reduce look-up time.

Example C++ snippet for a publisher with real-time QoS:

rclcpp::QoS qos(1);
qos.reliability(RMW_QOS_POLICY_RELIABILITY_BEST_EFFORT);
qos.durability(RMW_QOS_POLICY_DURABILITY_VOLATILE);
qos.deadline(rclcpp::Duration::from_seconds(0.01)); // 10 ms
qos.latency_budget(rclcpp::Duration::from_seconds(0.005)); // 5 ms budget

auto publisher = node->create_publisher<sensor_msgs::msg::LaserScan>("/scan", qos);

For subscribers, set matching profiles. Inconsistent QoS settings between publisher and subscriber can cause connection failures; always test with ros2 topic info to verify compatibility.

Testing and Optimization

Measuring Latency and Throughput

ROS2 provides several tools to evaluate real-time performance:

  • ros2 topic echo with the --csv flag logs message timestamps for offline analysis of jitter.
  • ros2 topic latency (from the ros2cli_common_extensions package) measures round-trip delay between two nodes.
  • Custom latency measurement using rclcpp::Clock stamped messages: publish a message with a synthetic timestamp and compute the delay on the subscriber side. This gives finer granularity than command-line tools.
  • ros2 doctor checks for common issues like QoS mismatches and network congestion.

Profile node CPU usage with perf or ros2 node info to identify bottlenecks. Pay special attention to callback execution time; if callbacks exceed a few microseconds, offload processing to a separate thread or reduce the message rate. Use std::chrono::high_resolution_clock inside callbacks to measure execution time, but be careful not to introduce dynamic allocation through the clock object.

Optimizing Message Transport

Large data streams – point clouds, images, high-rate IMU data – benefit from transport optimizations:

  • Zero-copy shared memory: Enable this when using a DDS that supports SHMEM (Fast DDS with FASTDDS_SHM_TRANSPORT=1, Cyclone DDS with shared memory enabled, or RTI Connext Pro with shared memory transport). This eliminates serialization overhead and reduces memory copies for large messages.
  • Reduce serialization overhead: Use flat message structures rather than deeply nested types. Avoid string fields in high-frequency messages; use fixed-length arrays or integer identifiers instead.
  • Adjust DDS history depth: With KEEP_LAST depth 1, the middleware only keeps the latest sample, minimizing memory and look-up time.
  • Use the Intra-Process Communication (IPC) feature when nodes run in the same process. Enable RCLCPP_INTRAPROCESS_ENABLED=1 and set rclcpp::IntraProcessSetting::Enable in your node. This bypasses DDS entirely for same-process communication, offering the lowest possible latency.

Advanced Real-Time Techniques

Real-Time Executor and Timer Scheduling

The StaticSingleThreadedExecutor is ideal for deterministic callback execution because it pre-allocates all required memory and avoids dynamic scheduling decisions. For control loops with periodic timers, create the timer with a fixed interval and attach it to a dedicated executor pinned to a real-time core:

auto executor = std::make_shared<rclcpp::executors::StaticSingleThreadedExecutor>();
executor->add_node(node);
// Pin the current thread to core 1 with SCHED_FIFO priority 90
pthread_t self = pthread_self();
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(1, &cpuset);
pthread_setaffinity_np(self, sizeof(cpu_set_t), &cpuset);
struct sched_param param;
param.sched_priority = 90;
pthread_setschedparam(self, SCHED_FIFO, &param);
executor->spin();

For multi-rate systems, use separate executors for each rate group, each pinned to a different core. This prevents a low-rate callback from blocking a high-rate one.

Integrating with Real-Time Linux (PREEMPT_RT)

When using the PREEMPT_RT kernel patch, ensure ROS2 nodes avoid blocking system calls within real-time threads. Common pitfalls include:

  • Dynamic memory allocation: Use pre-allocated pools in callbacks.
  • File I/O: Logging to disk should happen in a separate non-real-time thread.
  • Printf-style debugging: Use RCLCPP_DEBUG only in development; these calls can block on mutexes.
  • Priority inversion: Use priority inheritance mutexes (PTHREAD_PRIO_INHERIT) when protecting shared resources. ROS2’s DDS implementations typically handle this correctly, but custom mutexes must be configured explicitly.

Test your system under worst-case load with tools like cyclictest to measure maximum latency and ensure it stays below your deadlines.

Handling Multi-Rate and Mixed Criticality

Robotic systems often have sensors operating at different rates – a camera at 30 Hz, LiDAR at 10 Hz, IMU at 1 kHz. Use separate executors for each rate group, each pinned to a dedicated CPU core. Assign higher priorities to tasks with tighter deadlines (e.g., IMU processing at priority 95, control at 90, LiDAR at 80, non-critical logging at 50). For mixed-criticality, isolate safety-critical nodes in a separate real-time domain using middleware filters or the DDS partition mechanism. Consider using the ROS2 rclcpp_lifecycle framework to manage node state transitions (unconfigured → inactive → active) and ensure real-time components are only active when the system is ready.

Practical Use Case: Real-Time LiDAR Data Processing

Consider an autonomous mobile robot that must process a Velodyne LiDAR scan (300,000 points per second) and output a filtered point cloud with a maximum latency of 10 ms. The pipeline consists of:

  1. Driver node: Reads raw data from the LiDAR via an Ethernet socket and publishes raw packets (velodyne_msgs::msg::VelodyneScan) at the sensor’s native rate (10 Hz).
  2. Real-time filter node: Subscribes to raw packets, applies a passthrough filter (removing points above 2 meters height), and publishes filtered points (sensor_msgs::msg::PointCloud2).
  3. Localization node: Uses the filtered cloud for scan matching at 10 Hz.

To meet the 10 ms deadline:

  • Set QoS deadline to 10 ms on both raw and filtered topics.
  • Pin the driver and filter nodes to dedicated CPU cores with SCHED_FIFO priority 95.
  • Use shared memory transport to avoid serialization of 2 MB point cloud data. With Fast DDS, set FASTDDS_SHM_TRANSPORT=1 and configure the shared memory queue size to match the expected throughput.
  • Inside the filter callback, use pre-allocated fixed-size buffers. For example, allocate a pcl::PointCloud<pcl::PointXYZ> with reserve(300000) during node initialization, then reuse it in each callback with clear() (which does not reallocate capacity).
  • Avoid dynamic structures: do not use std::vector::push_back; instead track the number of valid points with a counter and insert via array indexing.

After deployment, monitor latency using a custom node that timestamps each filtered cloud and compares it to the original packet timestamp. Use ros2 topic latency to check round-trip times and perf stat -e context-switches to detect unnecessary preemptions. If deadlines are still missed, consider increasing the priority of the filter node or moving to a faster DDS implementation like Cyclone DDS with shared memory.

Conclusion

ROS2 provides a robust ecosystem for real-time robot data processing, combining distributed computing flexibility with the determinism required by modern autonomous systems. Success depends on selecting the right DDS implementation, configuring QoS policies correctly, and designing nodes with real-time constraints from the start. By using real-time kernels, priority scheduling, and zero-copy transport, developers can build systems that process high-rate sensor data with predictable latencies. As ROS2 evolves, support for time-sensitive networking (TSN) and tighter integration with hardware accelerators like FPGAs and GPUs will further extend its capabilities for mission-critical robotics.

For further reading, consult the official ROS2 real-time documentation (Real-Time Programming Tutorial), the DDS performance benchmarks from eProsima (Fast DDS), and the PREEMPT_RT project (Real-Time Linux Wiki). The ROS2 Real-Time Background article provides a deeper dive into the design philosophy behind the framework’s real-time support. For hands-on experimentation, the ROS2 demos repository includes real-time examples with executable code.