PCL/OpenNI tutorial 5: 3D object recognition (pipeline) - Revision history

Victorm at 23:26, 1 November 2015

2015-11-01T23:26:02Z

Victorm at 23:23, 1 November 2015

2015-11-01T23:23:58Z

Victorm at 09:22, 1 April 2015

2015-04-01T09:22:28Z

Victorm at 10:57, 23 February 2015

2015-02-23T10:57:49Z

Victorm at 10:36, 23 February 2015

2015-02-23T10:36:13Z

Victorm at 10:34, 23 February 2015

2015-02-23T10:34:44Z

Victorm at 12:51, 16 February 2015

2015-02-16T12:51:20Z

Victorm at 08:48, 13 February 2015

2015-02-13T08:48:10Z

Victorm at 10:24, 28 January 2015

2015-01-28T10:24:36Z

Victorm at 08:59, 30 December 2014

2014-12-30T08:59:01Z

@@ Line 37: / Line 37: @@
-Another possibility is to do it virtually, rendering 3D models of the objects (modelled beforehand with CAD software like [http://www.blender.org/ Blender]) and using the Z buffer, which contains the depth information, to simulate the input a depth sensor would give. The perks of doing this include: no segmentation step is necessary to get the object cluster, and the ground truth pose information will be 100% exact.
+Another possibility is to do it virtually, rendering 3D models of the objects (modelled beforehand with CAD software like [https://en.wikipedia.org/wiki/Blender_%28software%29 Blender]) and using the Z buffer, which contains the depth information, to simulate the input a depth sensor would give. The perks of doing this include: no segmentation step is necessary to get the object cluster, and the ground truth pose information will be 100% exact.
@@ Line 56: / Line 56: @@
 ==Getting a full model==
-Like I explained above, getting a full .pcd model for our object with a sensor requires some careful setup, good segmentation, normal computation and possibly resampling. On the other hand, if we use a 3D CAD model, it becomes much easier. A program like [http://www.blender.org/ Blender] (free and open source) can be used to design a model of the object, after taking some measurements. Programs like [http://meshlab.sourceforge.net/ MeshLab] can convert to and from many types of 3D formats. In this case, the one favored by PCL is binary [https://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] with normals (Blender can export to ASCII PLY, but I have sometimes encountered a processing error with those).
+Like I explained above, getting a full .pcd model for our object with a sensor requires some careful setup, good segmentation, normal computation and possibly resampling. On the other hand, if we use a 3D CAD model, it becomes much easier. A program like [https://en.wikipedia.org/wiki/Blender_%28software%29 Blender] (free and open source) can be used to design a model of the object, after taking some measurements. Programs like [https://en.wikipedia.org/wiki/MeshLab MeshLab] can convert to and from many types of 3D formats. In this case, the one favored by PCL is binary [https://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] with normals (Blender can export to ASCII PLY, but I have sometimes encountered a processing error with those).
 ===Raytracing===

@@ Line 25: / Line 25: @@
 ** [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Recognition.pdf 3D Object Recognition in Clutter with the Point Cloud Library]
 * '''Publication''':
-** [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
+** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
 </div>
@@ Line 56: / Line 56: @@
 ==Getting a full model==
-Like I explained above, getting a full .pcd model for our object with a sensor requires some careful setup, good segmentation, normal computation and possibly resampling. On the other hand, if we use a 3D CAD model, it becomes much easier. A program like [http://www.blender.org/ Blender] (free and open source) can be used to design a model of the object, after taking some measurements. Programs like [http://meshlab.sourceforge.net/ MeshLab] can convert to and from many types of 3D formats. In this case, the one favored by PCL is binary [http://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] with normals (Blender can export to ASCII PLY, but I have sometimes encountered a processing error with those).
+Like I explained above, getting a full .pcd model for our object with a sensor requires some careful setup, good segmentation, normal computation and possibly resampling. On the other hand, if we use a 3D CAD model, it becomes much easier. A program like [http://www.blender.org/ Blender] (free and open source) can be used to design a model of the object, after taking some measurements. Programs like [http://meshlab.sourceforge.net/ MeshLab] can convert to and from many types of 3D formats. In this case, the one favored by PCL is binary [https://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] with normals (Blender can export to ASCII PLY, but I have sometimes encountered a processing error with those).
 ===Raytracing===
@@ Line 66: / Line 66: @@
 <syntaxhighlight lang=Bash>pcl_mesh2pcd <input.ply> <output.pcd></syntaxhighlight>
-Here you can see the result of converting a mesh exported from Blender, the program's mascot, [http://en.wikipedia.org/wiki/Blender_%28software%29#Suzanne Suzanne]:
+Here you can see the result of converting a mesh exported from Blender, the program's mascot, [https://en.wikipedia.org/wiki/Blender_%28software%29#Suzanne Suzanne]:
@@ Line 182: / Line 182: @@
 ===ISS===
-Another keypoint detector available in PCL is the ISS (Intrinsic Shape Signatures) one. ISS is a local descriptor presented in 2009, which creates a view-independent signature of the local surface patch. An algorithm for choosing keypoints was also included to better fit the descriptor. The algorithm scans the surfaces and chooses only points with large variations in the [http://en.wikipedia.org/wiki/Principal_curvature principal direction] (the shape of the surface), which are ideal for keypoints.
+Another keypoint detector available in PCL is the ISS (Intrinsic Shape Signatures) one. ISS is a local descriptor presented in 2009, which creates a view-independent signature of the local surface patch. An algorithm for choosing keypoints was also included to better fit the descriptor. The algorithm scans the surfaces and chooses only points with large variations in the [https://en.wikipedia.org/wiki/Principal_curvature principal direction] (the shape of the surface), which are ideal for keypoints.
 You can use the associated PCL class this way:
@@ Line 424: / Line 424: @@
 ===Hough voting===
-This method is a bit more complex. It uses a technique known as [http://en.wikipedia.org/wiki/Hough_transform Hough transform], which was originally devised to perform line detection on 2D images, though it was later expanded to work with arbitrary shapes or higher dimensions. The method works with a voting procedure, with votes being cast on candidates that are found to be better suitable. If enough votes are accumulated for a given position in the space, then the shape is considered "found" there and its parameters are retrieved.
+This method is a bit more complex. It uses a technique known as [https://en.wikipedia.org/wiki/Hough_transform Hough transform], which was originally devised to perform line detection on 2D images, though it was later expanded to work with arbitrary shapes or higher dimensions. The method works with a voting procedure, with votes being cast on candidates that are found to be better suitable. If enough votes are accumulated for a given position in the space, then the shape is considered "found" there and its parameters are retrieved.
 The voting procedure is carried out in a parameter space, which would have 6 dimensions in case a full pose needed to be estimated (rotation plus translation). Because that would be computationally expensive, the method implemented in PCL uses only a 3D Hough space, and employs local Reference Frames (RFs) to account for the remaining 3 DoF. Its computation can then be divided in two steps.
@@ Line 858: / Line 858: @@
 ==Matching==
-Like with the local pipeline, once all the descriptors have been computed we have to perform a search for their nearest neighbors in the database. The [http://en.wikipedia.org/wiki/Taxicab_geometry Manhattan or L1 distance] is recommended over the [http://en.wikipedia.org/wiki/Euclidean_distance Euclidean or L2] one because it is more robust to occlusions.
+Like with the local pipeline, once all the descriptors have been computed we have to perform a search for their nearest neighbors in the database. The [https://en.wikipedia.org/wiki/Taxicab_geometry Manhattan or L1 distance] is recommended over the [https://en.wikipedia.org/wiki/Euclidean_distance Euclidean or L2] one because it is more robust to occlusions.
 Usually, more than one neighbor is retrieved, which means that the found object's pose is somewhere between the two. The next steps can help improving the accuraccy of the estimation.

@@ Line 103: / Line 103: @@
 ===Code===
-You can customize the snapshot generation process by making direct use of the <span style="color:#FF1493">"RenderViewsTesselatedSphere"</span> class, which is what the previous tool uses internally. The code is the following:
+You can customize the snapshot generation process by making direct use of the <span style="color:#FF1493">"RenderViewsTesselatedSphere"</span> class, which is similar to what the previous tool uses internally. The code is the following:
 <syntaxhighlight lang=CPP>#include <pcl/io/vtk_lib_io.h>

@@ Line 953: / Line 953: @@
 ==Pose refinement==
-The 6 DoF pose estimation can be refined with the use of the [http://en.wikipedia.org/wiki/Iterative_closest_point Iterative Closest Point] (ICP) algorithm. It will try to continuously realign the clouds to improve the transformation, until the termination condition is met (maximum iterations, and distance or error threshold).
+The 6 DoF pose estimation can be refined with the use of the [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Iterative_Closest_Point_.28ICP.29|Iterative Closest Point]] (ICP) algorithm. It will try to continuously realign the clouds to improve the transformation, until the termination condition is met (maximum iterations, and distance or error threshold). This is identical to performing [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Registration|registration]], only between input and model.
 ==Hypothesis verification==

@@ Line 869: / Line 869: @@
 ==Pose estimation==
-The object's position can be determined by computing and aligning the [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Computing_the_centroid|centroids]] of the clusters (the one in the scene, and the one in the snapshot). For the rotation, we must use the viewpoint information encoded in the descriptor. This still leaves the roll angle unresolved, as global descriptors are invariant to it. Now is when the Camera Roll Histogram ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH|CRH]]) that we should have stored next to the descriptor comes in handy. The angle can be obtained by aligning the current CRH with the stored one, completing the 6 DoF pose estimation.
+The object's position can be determined by computing and aligning the [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Computing_the_centroid|centroids]] of the clusters (the one in the scene, and the one in the snapshot). For the rotation, we must use the viewpoint information encoded in the descriptor (you can check the ground truth pose information that was saved with the matched snapshot). This still leaves the roll angle unresolved, as global descriptors are invariant to it. Now is when the Camera Roll Histogram ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH|CRH]]) that we should have stored next to the descriptor comes in handy. The angle can be obtained by aligning the current CRH with the stored one, completing the 6 DoF pose estimation.
 ===CRH===

← Older revision		Revision as of 12:51, 16 February 2015
Line 870:		Line 870:

	The object's position can be determined by computing and aligning the centroids of the clusters (the one in the scene, and the one in the snapshot). For the rotation, we must use the viewpoint information encoded in the descriptor. This still leaves the roll angle unresolved, as global descriptors are invariant to it. Now is when the Camera Roll Histogram ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH\|CRH]]) that we should have stored next to the descriptor comes in handy. The angle can be obtained by aligning the current CRH with the stored one, completing the 6 DoF pose estimation.		The object's position can be determined by computing and aligning the centroids of the clusters (the one in the scene, and the one in the snapshot). For the rotation, we must use the viewpoint information encoded in the descriptor. This still leaves the roll angle unresolved, as global descriptors are invariant to it. Now is when the Camera Roll Histogram ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH\|CRH]]) that we should have stored next to the descriptor comes in handy. The angle can be obtained by aligning the current CRH with the stored one, completing the 6 DoF pose estimation.
		+
		+	===CRH===
		+
		+	This is the code for retrieving the roll angle using CRH:
		+
		+	<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
		+	#include <pcl/common/centroid.h>
		+	#include <pcl/features/crh.h>
		+	#include <pcl/recognition/crh_alignment.h>
		+
		+	#include <iostream>
		+	#include <vector>
		+
		+	// A handy typedef.
		+	typedef pcl::Histogram<90> CRH90;
		+
		+	int
		+	main(int argc, char** argv)
		+	{
		+	// Clouds for storing the object's cluster and view.
		+	pcl::PointCloud<pcl::PointXYZ>::Ptr viewCloud(new pcl::PointCloud<pcl::PointXYZ>);
		+	pcl::PointCloud<pcl::PointXYZ>::Ptr clusterCloud(new pcl::PointCloud<pcl::PointXYZ>);
		+	// Objects for storing the CRHs of both.
		+	pcl::PointCloud<CRH90>::Ptr viewCRH(new pcl::PointCloud<CRH90>);
		+	pcl::PointCloud<CRH90>::Ptr clusterCRH(new pcl::PointCloud<CRH90>);
		+	// Objects for storing the centroids.
		+	Eigen::Vector4f viewCentroid;
		+	Eigen::Vector4f clusterCentroid;
		+
		+	// Read the trained view and the scene cluster from disk.
		+	if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *viewCloud) != 0)
		+	{
		+	return -1;
		+	}
		+	if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *clusterCloud) != 0)
		+	{
		+	return -1;
		+	}
		+
		+	// Note: here you would compute the CRHs and centroids of both clusters.
		+	// It has been omitted here for simplicity.
		+
		+	// CRH alignment object.
		+	pcl::CRHAlignment<pcl::PointXYZ, 90> alignment;
		+	alignment.setInputAndTargetView(clusterCloud, viewCloud);
		+	// CRHAlignment works with Vector3f, not Vector4f.
		+	Eigen::Vector3f viewCentroid3f(viewCentroid[0], viewCentroid[1], viewCentroid[2]);
		+	Eigen::Vector3f clusterCentroid3f(clusterCentroid[0], clusterCentroid[1], clusterCentroid[2]);
		+	alignment.setInputAndTargetCentroids(clusterCentroid3f, viewCentroid3f);
		+
		+	// Compute the roll angle(s).
		+	std::vector<float> angles;
		+	alignment.computeRollAngle(clusterCRH, viewCRH, angles);
		+
		+	if (angles.size() > 0)
		+	{
		+	std::cout << "List of angles where the histograms correlate:" << std::endl;
		+
		+	for (int i = 0; i < angles.size(); i++)
		+	{
		+	std::cout << "\t" << angles.at(i) << " degrees." << std::endl;
		+	}
		+	}
		+	}</syntaxhighlight>
		+
		+	This will return a list of most probable roll angles. If you want, you can instead use the [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_alignment.html#a32c40356d56e22d48eccce5e27e436f2 align()] function, which calls <span style="color:#FF1493">"computeRollAngle()"</span> internally. It will return a list of transformations that include the translation (computed from the difference between the centroid's coordinates) and the roll angles, but not the yaw or pitch (you have to get those with pose estimation).
		+
		+	<div style="background-color: #F8F8F8; border-style: dotted;">
		+	* '''Input''': Points (input and target), Centroids (input and target), CRHs (input and target)
		+	* '''Output''': Roll angles, [Transformations]
		+	* '''Publication''':
		+	** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
		+	* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_alignment.html pcl::CRHAlignment]
		+	* [http://robotica.unileon.es/~victorm/PCL_roll_estimation.tar.gz Download]
		+	</div>
		+

	=Postprocessing=		=Postprocessing=

← Older revision		Revision as of 08:48, 13 February 2015
Line 880:		Line 880:

	==Hypothesis verification==		==Hypothesis verification==
		+
		+	''(Work in progress)''

@@ Line 50: / Line 50: @@
 <div style="background-color: #F8F8F8; border-style: dotted;">
-* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php#vfh-recognition Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
+* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
 </div>
@@ Line 330: / Line 330: @@
 <div style="background-color: #F8F8F8; border-style: dotted;">
-* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php#correspondence-grouping 3D Object Recognition based on Correspondence Grouping]
+* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
 * '''API''':
 ** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
@@ Line 416: / Line 416: @@
 * '''Input''': Points (model), Points (scene), Correspondences, Minimum cluster size, Consensus set resolution
 * '''Output''': Clustered correspondences, [Transformation]
-* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php#correspondence-grouping 3D Object Recognition based on Correspondence Grouping]
+* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
 * '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_geometric_consistency_grouping.html pcl::GeometricConsistencyGrouping]
 * [http://robotica.unileon.es/~victorm/PCL_local_pipeline_GC_grouping.tar.gz Download]
@@ Line 492: / Line 492: @@
 * '''Input''': Points, Normals, Search radius, [Tangent radius], [Margin array size], [Margin threshold], [Find holes], [Hole size threshold], [Steepness threshold]
 * '''Output''': Reference Frames
-* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php#correspondence-grouping 3D Object Recognition based on Correspondence Grouping]
+* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
 * '''Publication''':
 ** [http://vision.deis.unibo.it/LRF/LRF_repeatability_ICCV2011.pdf On the repeatability of the local reference frame for partial shape matching] (Alioscia Petrelli and Luigi Di Stefano, 2011)
@@ Line 577: / Line 577: @@
 * '''Input''': Points (model), Points (scene), Correspondences, Reference Frames (model), Reference Frames (scene), Hough bin size, Hough threshold, [Use distance weight], [Use interpolation]
 * '''Output''': Clustered correspondences, [Transformation]
-* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php#correspondence-grouping 3D Object Recognition based on Correspondence Grouping]
+* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
 * '''Publication''':
 ** [http://vision.deis.unibo.it/fede/papers/psivt10.pdf Object recognition in 3D scenes with occlusions and clutter by Hough voting] (Federico Tombari and Luigi Di Stefano, 2010)
@@ Line 666: / Line 666: @@
 * '''Output''': Aligned model, [Transformation]
 * '''Tutorials''':
-** [http://pointclouds.org/documentation/tutorials/alignment_prerejective.php#alignment-prerejective Robust pose estimation of rigid objects]
+** [http://pointclouds.org/documentation/tutorials/alignment_prerejective.php Robust pose estimation of rigid objects]
-** [http://pointclouds.org/documentation/tutorials/template_alignment.php#template-alignment Aligning object templates to a point cloud]
+** [http://pointclouds.org/documentation/tutorials/template_alignment.php Aligning object templates to a point cloud]
 * '''Publication''':
 ** [http://personal.lut.fi/users/joni.kamarainen/downloads/publications/icra2013.pdf Pose Estimation using Local Structure-Specific Shape and Appearance Context] (Anders Glent Buch et al., 2013)
@@ Line 863: / Line 863: @@
 <div style="background-color: #F8F8F8; border-style: dotted;">
-* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php#vfh-recognition Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
+* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
 </div>