Sensing

The “Sensing” section serves as an introduction to a broad spectrum of sensory technologies critical for robotic perception, navigation, and interaction. It outlines tools and techniques that empower robots to effectively sense their surroundings and make informed decisions.

This section dives into various sensing modalities such as GPS modules, fiducial markers, radar, thermal cameras, stereo vision, and many more. It provides details on their principles of operation, implementation, and practical use cases, along with helpful troubleshooting tips. Whether you’re working on autonomous navigation, precise localization, object detection, or environmental mapping, this section is designed to equip you with the foundational knowledge and resources to integrate these sensors into your projects.

Key Subsections and Highlights

  • Adafruit GPS: Discusses the Adafruit Ultimate GPS module for precise localization, including its features, technical specifications, and usage tips.

  • Apple Vision Pro: Provides an overview on integrating Apple Vision Pro with robotics, covering object tracking, spatial coordinate frames, and ROS communication.

  • AprilTags: Introduces AprilTags as a fiducial marker system for visual identification and 6D pose estimation. Explains their use in robotics and potential pitfalls.

  • Azure Block Detection: Outlines the use of the Azure Kinect for object detection, with applications like Jenga block assembly. Explains the step-by-step detection pipeline.

  • Camera Calibration: Emphasizes the importance of calibrating cameras for minimizing errors and improving vision system accuracy. Includes references to key calibration resources.

  • Camera-IMU Calibration using Kalibr: Details the Kalibr library for simultaneous IMU and camera calibration, including example setups and tips for accurate calibration.

  • Computer Vision Considerations: Highlights key considerations when deploying computer vision in robotics, including lighting, frame rates, calibration, and error mitigation.

  • Delphi ESR Radar: Provides an overview of Delphi’s ESR radar for detecting objects and estimating their range, speed, and position.

  • DWM1001 UltraWideband Positioning System: Covers the setup and calibration of the DWM1001 UWB system for accurate indoor positioning.

  • Fiducial Markers: Compares various fiducial marker systems like ArUco, AprilTags, and STag, listing their pros, cons, and ideal use cases.

  • Hand-Eye Calibration: Provides a tutorial for estimating the frame transformation between an image frame and an operating frame using ROS2 and Aruco markers.

  • Intel Realsense: Introduces Intel RealSense cameras and details SDK installation, ROS integration, calibration, and tuning methods.

  • OpenCV Stereo Vision Processing: Introduces OpenCV libraries for stereo vision, including camera calibration and 3D triangulation.

  • Perception via Thermal Imaging: Discusses strategies to implement key steps in a robotic perception pipeline using thermal cameras, including depth estimation and metric recovery.

  • Photometric Calibration: Explains the need for calibrating camera sensors to accurately map light intensity to pixel values, and methods to achieve this.

  • Point Cloud Library, 3D Sensors and Applications: Discusses PCL’s features for processing 3D point clouds and its applications in object detection, segmentation, and mapping.

  • RTK GPS: Explains how to achieve centimeter-level accuracy using RTK GPS systems, along with practical lessons and setup guidance.

  • Reducing Sensor Noise in Thermal or Visual Imaging sensors: Explains various techniques for reducing noise in thermal and visual imaging sensors, including filtering and post-processing methods.

  • Robot-Centric Elevation Mapping: Explains robot-centric elevation mapping using the Grid Map library to create 2.5D maps centered around the robot, accounting for pose uncertainty.

  • Robotic Total Station (Leica TS16): Discusses the use of robotic total stations for high-precision 3D positioning and their applications in surveying and robotics.

  • Robotics with the Microsoft Hololens2: Introduces the Microsoft HoloLens 2 as an AR headset for robotics applications, focusing on accessing its onboard sensors (RGB, Greyscale, Depth, and IMU) using a Unity-Python API.

  • Speech Recognition: Explores speech recognition as a robotic interface, including offline and online solutions, wakeword detection, and speech synthesis.

  • STag: Presents the STag fiducial marker system for stable pose estimation, with details on implementation and experimental results.

  • Thermal Cameras: Examines the use of thermal cameras in robotics, including types of thermal cameras, calibration techniques, and debug tips.

  • Tracking vehicles using a static traffic camera: Describes a system for extracting vehicle trajectories using static traffic cameras, incorporating detection, tracking, and homography estimation.

Resources