Benchmark dataset for evaluation of range-based people tracker classifiers in mobile robots
SciCrunch! reference: Range-based people tracker classifiers Benchmark Dataset, RRID:SCR_015743.
This data report summarizes a benchmark dataset which can be used to evaluate the performance of different approaches for detecting and tracking people by using LIDAR sensors. Information contained at the dataset is specially suitable to be used as training data for neural network-based classifiers.
Further information at Álvarez-Aparicio et al. (2017).
Contents
[hide]Materials and methods
The following section describes the materials (shown in Figure 1) used to gather data, which include: a certified study area, an autonomous robot with an on-board LIDAR sensor, and a real-time location system (RTLS) to obtain ground-truth data about person location. Recorded data include location estimates calculated by two people trackers, LD and PeTra, also described below. Finally, the recording procedure used to build the dataset is explained.
Leon@Home Testbed
Data have been gathered at Leon@Home Testbed. This is a Certified Testbed of the European Robotics league (ERL). Its main purpose is to benchmark service robots in a realistic home environment. Our testbed is made up of four parts, shown in Figure 1B: a mock-up apartment, a control zone with direct vision (glass wall) into the apartment, a small workshop, and a larger development zone, where researchers work.
Leon@Home Testbed is located on the second floor of the Módulo de Investigación en Cibernética (Building for Research in Cybernetics) on the Vegazana Campus of the University of León (Spain). The apartment is a single bedroom mock-up home built in an 8 m × 7 m space. Figure 1C shows a plan of the apartment. 60 cm high walls divide it into a kitchen, living room, bathroom, and bedroom. The furniture (Figures 1E,F) has been chosen to test different robot abilities. For instance, the kitchen cabinets all have different types of handles.
Orbi-one
Orbi-One (Figure 1A) is an assistant robot manufactured by Robotnik. It has several sensors, among them, a RGBD camera, a LIDAR sensor, and an inertial unit. It can operate a manipulator arm attached to its torso and has a wheeled base for moving around the room. Orbi-One includes a wireless access point, which allows WiFi communications with other robots and computers.
The software to control the robot hardware is based on a ROS framework. ROS is basically a set of libraries for robotics similar to operating system services, providing hardware abstraction for sensors and actuators, low-level device control, and inter-process communication. Computation takes place in processes named Nodes, which can receive and send Messages. Nodes publish Messages into information buffers called Topics.
KIO RTLS
In order to acquire ground-truth data about person location in the study area, we need an RTLS for indoor environments. The KIO RTLS commercial solution by Eliko has been used. KIO is a precise RTLS for tracking any object in 2- or 3-dimensional space. The Ultra Wideband technology enables to micro-position objects through obstructions. KIO also works in non-line-of-sight conditions and both indoors and outdoors.
KIO comes in two main configurations. The Regular Cell configuration guarantees a reliable accuracy of ±30 cm, according to the manufacturer’s specifications. The Small Cell configuration is designed for location-critical applications and provides reliable ±5 cm accuracy, according to the manufacturer’s specifications. Calibration done by the authors of this paper on the mock-up apartment shows that the error is higher in some areas, and lower in others, but on average, the claims of the manufacturer are correct.
KIO calculates the position of a mobile transceiver, called a Tag. In order to do so, KIO uses radio beacons, called Anchors, distributed in known positions in the surroundings. Figure 1D shows a KIO anchor. KIO tags are the same size and must be placed on-board the tracking subject, in our case people. The red dots in Figure 1C show the location of the six anchors used in these experiments. They are placed on the ceiling. The distribution of the anchors has been chosen following the method shown in Guerrero-Higueras et al. (2017).
Leg Detector (LD)
LD is a ROS package, which takes messages published by a LIDAR sensor as input and uses a machine-learning-trained classifier to detect groups of laser readings as possible legs. The code is available in a public repository, but is unsupported at this time.
LD publishes the location for the individual legs. It can also attempt to pair the legs and publish their average as an estimate of where the center of a person is. LD may optionally also publish visualization marker messages to indicate where detections happened.
PeTra
PeTra is a person-tracker tool for detecting and tracking, developed by the Robotics Group at the University of León. The system is based on a Convolutional Neural Network (CNN) using a configuration based on the U-Net architecture by Ronneberger et al. (2015).
The system performs the following steps in real time:
1. First, the data provided by the LIDAR sensor are processed to build a two dimensional occupancy map centered around the robot. This occupancy map is represented as a binary matrix, where 1s denote positions where the LIDAR scan found an obstacle, and 0s denote positions where the LIDAR scan either went through without detecting any obstacle or did not go through that position.
2. Then, the occupancy map is relayed to the network as input data. The network produces a second occupancy map representing the zones where legs have been detected.
3. Finally, center of mass calculations return the location of persons. PeTra also publishes locations for the individual legs and Marker messages for visualization.
PeTra system have been trained using the following data: npy_train_test_globales.tar.gz
Recording procedure
The data were gathered in 14 different situations. In all of them, Orbi-One stood still as one or more people, carrying a KIO tag, moved around it. Three different locations for Orbi-One were defined (see Figure 1C) resulting in 42 scenarios (14 situations × 3 Orbi-One locations). Figure 2 shows the 14 different recognition scenarios recorded. These scenarios have been chosen according to different situations that may occur in robotics competitions such as ERL or RoboCup.
Datasets Scenarios Fig. 2
A rosbag file was created for each scenario (except for situations 3, 12, and 13 where 3 rosbag files where recorded, for situation 4 where 4 rosbag files where recorded, and for situation 9 where 5 rosbag files where recorded), recording LIDAR sensor measurements, location estimates from PeTra and LD, locations from KIO RTLS, and other useful data. Specifically, the following data were included in the rosbag files:
- LIDAR sensor data. Data from LIDAR sensors are provided as ROS LaserScan Messages, which include, among other information, the following: acquisition time of the first ray in the scan, start/end angle of the scan, angular distance between measurements, and range data.
- PeTra location estimates provided as ROS PointStamped Messages, which include a position [x, y, z] and a timestamp.
- Location estimates calculated by LD. It publishes data for individual legs (as ROS PositionMeasurementArray Messages). It also attempts to pair the legs and publishes their average as an estimate of where the center of a person is as a ROS PositionMeasurement Message.
- Locations provided by KIO RTLS also provided as ROS PointStamped Messages.
- Messages from /map, /odom, and /tf ROS topics, which include map information, odometry of the robot base, and transformation information, respectively.
Differents versions of the dataset are enumerated below.
v1.0 [Nov-2017]
As a result of applying the recording method explained above, a first version of the dataset have been released. It includes measures for the scenarios defined at Fig. 2.:
| Location 1. Kitchen | Location 2. Living room | Location 3. Bedroom |
|---|---|---|
|
|
|
v2.0 [Feb-2018]
As a result of applying the recording method explained above, a second version of the dataset has been released. It includes measures for the scenarios defined at Fig. 2.:
| Location 1. Kitchen | Location 2. Living room | Location 3. Bedroom |
|---|---|---|
|
|
|
v3.0 [Dec-2018]
As a result of applying the recording method explained above, a third version of the dataset has been released. It includes measures for 2 scenarios (5,10) defined in Fig. 2. In those rosbags, the robot OrbiOne is moving at recording time:
| Location 1. Kitchen | Location 2. Living room | Location 3. Bedroom |
|---|---|---|
|
|
|
Datasets Scenarios Fig. 3
A rosbag file was created for each scenario presented in Fig.3 in order to evaluate the people tracking. It includes measures for the three scenarios in three different locations:
| Location 1. Kitchen | Location 2. Living room | Location 3. Hall |
|---|---|---|
|
|
