PCL/OpenNI tutorial 2: Cloud processing (basic) - Revision history

Victorm at 19:47, 18 November 2015

2015-11-18T19:47:11Z

Victorm at 16:51, 5 November 2015

2015-11-05T16:51:09Z

Victorm at 11:03, 2 November 2015

2015-11-02T11:03:09Z

Victorm at 23:05, 1 November 2015

2015-11-01T23:05:50Z

Victorm at 23:04, 1 November 2015

2015-11-01T23:04:43Z

Victorm at 23:02, 1 November 2015

2015-11-01T23:02:09Z

Victorm at 23:01, 1 November 2015

2015-11-01T23:01:21Z

Victorm at 23:00, 1 November 2015

2015-11-01T23:00:38Z

Victorm at 11:22, 23 February 2015

2015-02-23T11:22:53Z

Victorm at 11:16, 23 February 2015

2015-02-23T11:16:03Z

@@ Line 263: / Line 263: @@
 	Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();
-	// Set a rotation around the Z axis.
+	// Set a rotation around the Z axis (right hand rule).
-	float theta = M_PI / 2; // 180 degrees.
+	float theta = 90.0f * (M_PI / 180.0f); // 90 degrees.
 	transformation(0, 0) = cos(theta);
 	transformation(0, 1) = -sin(theta);

← Older revision		Revision as of 16:51, 5 November 2015
Line 294:		Line 294:


−	Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.	+	Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object. Another important note: the function <span style="color:#FF1493">"addCoordinateSystem()"</span> with those parameters seen above is marked as deprecated in new versions of PCL. From these on you will have to use something like this:
		+
		+	<syntaxhighlight lang=CPP>viewer.addCoordinateSystem(1.0, "reference", 0);</syntaxhighlight>

	<div style="background-color: #F8F8F8; border-style: dotted;">		<div style="background-color: #F8F8F8; border-style: dotted;">

@@ Line 282: / Line 282: @@
 	viewer.addPointCloud(transformed, colorHandler, "transformed");
 	// Add 3D colored axes to help see the transformation.
-	viewer.addCoordinateSystem(1.0, "reference", 0);
+	viewer.addCoordinateSystem(1.0, 0);
 	while (!viewer.wasStopped())

@@ Line 1,149: / Line 1,149: @@
 A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.
-Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://developer.nvidia.com/about-cuda CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.
+Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://en.wikipedia.org/wiki/CUDA CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.
 Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

@@ Line 240: / Line 240: @@
 ==Matrix transformations==
-Clouds can be directly manipulated with [http://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.
+Clouds can be directly manipulated with [https://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.
 <syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
@@ Line 364: / Line 364: @@
 ==Removing NaNs==
-The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [http://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:
+The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [https://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:
 <syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
@@ Line 465: / Line 465: @@
 ==Computing the centroid==
-The [http://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.
+The [https://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.
 The centroid of a cloud can be computed with a single function call:
@@ Line 511: / Line 511: @@
 ==Normal estimation==
-As you may remember from geometry class, the [http://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.
+As you may remember from geometry class, the [https://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.
 I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).
@@ Line 645: / Line 645: @@
 ==''k''-d tree==
-A [http://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).
+A [https://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).
 It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.
@@ Line 730: / Line 730: @@
 ==Octree==
-Like the ''k''-d tree, the [http://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
+Like the ''k''-d tree, the [https://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
 voxel resolution. That way, we can selectively give more resolution to certain zones.
@@ Line 1,100: / Line 1,100: @@
 ===Statistical===
-The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [http://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).
+The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [https://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).
 <syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
@@ Line 1,246: / Line 1,246: @@
 ===Upsampling===
-Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [http://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!
+Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [https://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!
-PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:
+PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:
 <syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
@@ Line 1,308: / Line 1,308: @@
 ==Surface smoothing==
-As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.
+As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.
 Surface smoothing is easily done with the following code:

@@ Line 19: / Line 19: @@
 First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.
-Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from [http://www.boost.org/ Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.
+Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.
 ==PCD files==

@@ Line 11: / Line 11: @@
 Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.
-Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [http://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.
+Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.
 Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

@@ Line 9: / Line 9: @@
 All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"
-Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [http://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.
+Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.
 Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [http://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

← Older revision		Revision as of 11:22, 23 February 2015
Line 1,305:		Line 1,305:

	=Reconstruction=		=Reconstruction=
−
−	~~To learn about how to use PCL to build a smooth, parametric surface representation from a point cloud, you can read the following:~~
−
−	~~<div style="background-color: #F8F8F8; border-style: dotted;">~~
−	* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/bspline_fitting.php Fitting trimmed B-splines to unordered point clouds]
−	~~</div>~~
−

	==Surface smoothing==		==Surface smoothing==

@@ Line 1,311: / Line 1,311: @@
 * '''Tutorial''': [http://pointclouds.org/documentation/tutorials/bspline_fitting.php Fitting trimmed B-splines to unordered point clouds]
 </div>
 ==Surface smoothing==