Simultaneous Localization And Mapping (SLAM) is a well-studied problem in robotics, with many existing approaches and solutions. Although indoor SLAM is a mostly solved problem, SLAM in outdoor, unstructured environments is significantly more challenging, with no well-established solutions. This project evaluated the performance of four state-of-the-art open-source SLAM algorithms across three different outdoor environments to learn how well (or poorly) each performed and how much effort was needed to adapt these algorithms for the environment.
We evaluated three different SLAM algorithms: Lightweight and Ground-Optimized LIDAR Odometry and Mapping (LeGO-LOAM), Oriented features from accelerated segment test and Rotated Binary robust independent elementary features for SLAM (ORB-SLAM2), and Google Cartographer. Each algorithm was tested across three different environments: SwRI’s test track (wide-open with minimal features), SwRI’s campus (urban area with dense buildings), and SwRI’s off-road trails.
The algorithm performance was compared using a medium length route (approximately 1.6 km) for each environment. The comparison showed that none of the algorithms worked particularly well, with most algorithms accumulating at least 10 m of error after a single loop. LeGO-LOAM performed the worst, accumulating between 150 m and 1,010 m error, while Cartographer performed the best with 9.79 m to 118.51 m of accumulated error. Performance varied across different environments, but not as expected; in several cases wide-open areas proved worse than off-road areas. Cartographer performed the worst in wide-open spaces, while LeGO-LOAM and ORB-SLAM2 performed most poorly on the off-road trails.
Our primary conclusion is that none of the approaches we evaluated worked “out-of-the-box” and took considerable effort to tune and make functional. After this tuning, we were able to get two out of three algorithms to produce decent results (Figure 1 shows performance at SwRI’s test track) for some of the three environments, but no algorithm or set of parameters worked reliably in all environments.