robotica.unileon.es - User contributions [en]

TFM-Nao-Goalkeeper

2016-04-20T18:54:03Z

Victorm:

* '''Project name:''' TFM-Nao-Goalkeeper
* '''Dates:''' October 2010 - July 2011
* '''Degree:''' MSc
* '''Author:''' [[User:Victorm | Víctor Rodríguez]]
* '''Contact:''' vrodm [at] unileon [dot] es
* '''Tags:''' ai, state machine, robocup, goalkeeper
* '''Technologies:''' nao, naoqi, robocup, c++, cmake, svn, bica
* '''Status:''' Finished (used in [http://www.robocup2011.org RoboCup 2011])

=Nao articles=

Mostly deprecated by now, but here you have:

[[Nao tutorial 1: First steps]]

[[Nao tutorial 2: First module]]

[[Nao troubleshooting]]

Activities

2016-03-01T09:35:17Z

Victorm:

=Last Events=

==3D Scanning and Printing==

During the last two weeks Francisco Sedano gave a brief introduction to 3D prototyping technologies that can be applied to a wide variety of engineering applications. The training gave the students the knowledge to develop and use 3D scanning and 3D printing technologies.

[[Image:20160218_195229.jpg|center|border|300px]]

[http://robotica.unileon.es/mediawiki/documents/escaneado_triptico.pdf Scanning Course]

[http://robotica.unileon.es/mediawiki/documents/impresoras_3D.pdf 3D Printing Course]

==Francisco Lera's PhD dissertation==

Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

==RoCKIn 2015 (Lisboa) ==

[http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015], the second competition event in the EU-funded robotics project RoCKIn, was held in Lisbon (Portugal), 21-23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

It was a joint event with the [https://web.fe.up.pt/~robot2015/ ROBOT'2015], the Second Iberian Robotics Conference where we also presented [[a paper]]

==RoCKIn Event 2014 (Toulouse)==

(main article of this event [[RoCKInChallengeToulouse | here]])

==RoCKIn Camp 2014 (Rome)==

(main article of this event [[RoCKIn2014 | here]])

The team was in Rome testing the MYRABot robot. There, we learned different topics about vision, grasping, and natural language understanding. We tested the MB+ (MYRABot+) robot, a modification of MYRABot. Other robots were introduced in this competition, such as REEM from PAL Robotics.

[[Image:MBPAL1.JPG|center|border|300px]]

==Hack For Good 2013==

Proposals for [http://hackforgood.net HackForGood] can be found [[proposals-hackforfood-2013 | here]].

=Seminars=

* European Robotics Week Conference [[http://www.eu-robotics.net/eurobotics-week/events-2013/low-cost-service-robots-ule-watermelon-project-for-rockinhome-challenge.html | Low Cost Robots for RoCKIn Challenge ]]November 27th, 2013

* Course [[ProgramacionRobots2012|Introduction to Mobile Robot Programming (4th edition)]]. October 22nd to 26th , 2012

* Conference ''"Augmented reality for elderly medication control using a mobile robot"'' at the Curso de Verano [http://catedratelefonica.unileon.es/2012/07/09/participacion-de-la-catedra-en-el-curso-de-verano-envejecimiento-activo-una-aproximacion-a-la-realidad-actual/ Envejecimiento Activo].

[[Image:Foto-Curso-Verano-2012.jpeg|center|300px]]

* Conference ''"The role of edutainment robotics in mechatronics research and teaching"'' by [http://www.ipb.pt/~jllima/ José Luis Lima] and José Gonçalves from the [http://www.ipb.pt Politechical Institute of Bragança], on November 4, 2011.

[[Image:CharlaIPB-2011.jpg|center|300px]]

* Course [[ProgramacionRobots2011 | Mobile Robot Programming, medium level (1st edition)]]. March 2011.

* Seminar on BICA programming in the SPITeam for the RoboCup2011. Instructors: Francisco Martín and Carlos E. Agüero from [http://www.robotica-urjc.es URJC Robotics Group].

<wikiflv width="300" height="250" logo="true">/videos/CursoBica.flv</wikiflv>

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (3rd edition)]]. December 9 to 23, 2010.

[[Image:16122012.JPG|center|300px]]

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (2nd edition)]]. March 8 to 12, 2010.

<wikiflv width="300" height="250" logo="true">/videos/videoSumo20100312.flv</wikiflv>

* Course [http://roboticaleon2009.googlepages.com/ Mobile Robot programming]. July 1 to 7, 2009.

[[Image:Carrera.jpg|center|300px]]

* [http://www.compcog.org/ CompCog] Workshop ESF on Held in the University of León, May 26 to 27, 2009.

[[Image:Compcog09-leon.jpg|center|300px]]

* Introduction seminar to Nao programming (Instructor [http://www.robotica-urjc.es Dr. Francisco Martín Rico]),

[[Image:Workshop-paco09.jpg|center|300px]]

=In the Press=

* Article in Diario de León about the [http://www.diariodeleon.es/noticias/sociedad/orbione-amigo-robot-ayuda-casa_1048422.html OrbiOne robot]. We made the front page! (February 24th, 2016)

[[Image:PortadaDiarioLeon2016.jpg|center|300px]]

* [http://www.diariodevalladolid.es/noticias/valladolid/villanueva-augura-empresas-innoven-creceran-triple-proximos-cinco-anos_14514.html Award ''Innovadores Castilla y Leon 2015''] (El Mundo, 2015)

*[http://www.elmundo.es/economia/2014/12/31/54a2ee5b22601d42688b4580.html?intcmp=ULNOH002 Robots que medican y chalan], coverage by national newspaper of the group's activities (El Mundo, Dec, 2014)

* [http://www.diariodeleon.es/noticias/leon/premio-tic_845853.html Innova Award] to the best 2014 IT project, given by local newspaper (Diario de León).

* Article about Fernando's work on mobile robots [http://www.fernando.casadogarcia.es/innova.jpg Suplemento Innova (Diario de León)]

[[Image:DL-FernandoCasado.jpg|center|300px]]

* Juan Felipe was awarded the [http://www.diariodeleon.es/noticias/leon/juan-felipe-garcia_694743.html Young Talent 2012] title by Diario de León.
* Article about using mobile robots in [http://www.la-cronica.net/2011/02/28/leon/robots-moviles-para-asistir-a-personas-dependientes-113536.htm La Crónica].

* Conference on social robotics applied to handicapped people. Organized by CRE:

# Information about the event at [http://www.imserso.es/cresanandres_01/auxiliares/actualidad/novedades/2011/enero/IM_042486?dDocName=IM_042486 CRE's webpage].

# Press note at [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577647 Diario de León].

* Press Note about [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577402 Juan's PhD Viva].

[[Image:FotoDiarioLeon2011.jpg|center|300px]]

* Yet another info in the local newspaper, about a project on [http://www.diariodeleon.es/noticias/noticia.asp?pkid=571538 robotics for security].

* The same one in the local TV:

<videoflash>Z-lNsL-o8UY</videoflash>

* Another review in the local newspapers: [http://www.leonoticias.com/frontend/leonoticias/Dos-Robots-leoneses-Acudiran-A-La-Copa-De-Europa-De-Futb-vn45312-vst384 Leonnoticias], and [http://www.la-cronica.net/2010/03/11/leon/ingenieros-de-la-ule-estaran-en-la-champions-de-futbol-de-robots-72130.htm La Crónica], about the team entering the [http://www.robocup-mediterranean-open.org/ Mediterranean Open] (Rome, March 18 to 20, 2010).

* Interview in the local newspaper (Diario de León): [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495839 first one] and [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495838 second one].

[[Image:FotoDiarioLeon.jpg|center|300px]]

= RoboCup =

* We were proud members of the Spanish Team ([https://twitter.com/spiteam_robot SPITeam]) in the RoboCup Standard Platform providing a goalkeeper. SPITeam was a joint effort of the [robotics group] of the Rey Juan Carlos University, Rovira y Virgili University and Universidad de León. Last Team Description paper can be found [http://www.gsyc.es/~fmartin/SPiTeamApplication.pdf here]. Our last competition entrance was in [http://www.robocup2011.org/en/content.asp?PID={41EB0A0E-7EAF-4365-98F7-1D951C9B8D88}&PageFn=Teams Instambul 2011]

* We were previously also proud members of the [http://www.teamchaos.es TeamChaos] RoboCup SPL team. We have contributed the goalie to the team for the [http://www.robocup2009.org/ RoboCup 2009] and [http://www.robocup-german-open.de/en German Open 2009].

[[Image:Teamchaos-08.JPG|center|300px]]

Activities

2016-02-23T13:24:00Z

Victorm:

=Last Events=

==3D Scanning and Printing==

During the last two weeks Francisco Sedano gave a brief introduction to 3D prototyping technologies that can be applied to a wide variety of engineering applications. The training gave the students the knowledge to develop and use 3D scanning and 3D printing technologies.

[[Image:20160218_195229.jpg|center|border|300px]]

[http://robotica.unileon.es/mediawiki/documents/escaneado_triptico.pdf Scanning Course]

[http://robotica.unileon.es/mediawiki/documents/impresoras_3D.pdf 3D Printing Course]

==Francisco Lera's PhD dissertation==

Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

==RoCKIn 2015 (Lisboa) ==

[http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015], the second competition event in the EU-funded robotics project RoCKIn, was held in Lisbon (Portugal), 21-23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

It was a joint event with the [https://web.fe.up.pt/~robot2015/ ROBOT'2015], the Second Iberian Robotics Conference where we also presented [[a paper]]

==RoCKIn Event 2014 (Toulouse)==

(main article of this event [[RoCKInChallengeToulouse | here]])

==RoCKIn Camp 2014 (Rome)==

(main article of this event [[RoCKIn2014 | here]])

The team was in Rome testing the MYRABot robot. There, we learned different topics about vision, grasping, and natural language understanding. We tested the MB+ (MYRABot+) robot, a modification of MYRABot. Other robots were introduced in this competition, such as REEM from PAL Robotics.

[[Image:MBPAL1.JPG|center|border|300px]]

==Hack For Good 2013==

Proposals for [http://hackforgood.net HackForGood] can be found [[proposals-hackforfood-2013 | here]].

=Seminars=

* European Robotics Week Conference [[http://www.eu-robotics.net/eurobotics-week/events-2013/low-cost-service-robots-ule-watermelon-project-for-rockinhome-challenge.html | Low Cost Robots for RoCKIn Challenge ]]November 27th, 2013

* Course [[ProgramacionRobots2012|Introduction to Mobile Robot Programming (4th edition)]]. October 22nd to 26th , 2012

* Conference ''"Augmented reality for elderly medication control using a mobile robot"'' at the Curso de Verano [http://catedratelefonica.unileon.es/2012/07/09/participacion-de-la-catedra-en-el-curso-de-verano-envejecimiento-activo-una-aproximacion-a-la-realidad-actual/ Envejecimiento Activo].

[[Image:Foto-Curso-Verano-2012.jpeg|center|300px]]

* Conference ''"The role of edutainment robotics in mechatronics research and teaching"'' by [http://www.ipb.pt/~jllima/ José Luis Lima] and José Gonçalves from the [http://www.ipb.pt Politechical Institute of Bragança], on November 4, 2011.

[[Image:CharlaIPB-2011.jpg|center|300px]]

* Course [[ProgramacionRobots2011 | Mobile Robot Programming, medium level (1st edition)]]. March 2011.

* Seminar on BICA programming in the SPITeam for the RoboCup2011. Instructors: Francisco Martín and Carlos E. Agüero from [http://www.robotica-urjc.es URJC Robotics Group].

<wikiflv width="300" height="250" logo="true">/videos/CursoBica.flv</wikiflv>

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (3rd edition)]]. December 9 to 23, 2010.

[[Image:16122012.JPG|center|300px]]

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (2nd edition)]]. March 8 to 12, 2010.

<wikiflv width="300" height="250" logo="true">/videos/videoSumo20100312.flv</wikiflv>

* Course [http://roboticaleon2009.googlepages.com/ Mobile Robot programming]. July 1 to 7, 2009.

[[Image:Carrera.jpg|center|300px]]

* [http://www.compcog.org/ CompCog] Workshop ESF on Held in the University of León, May 26 to 27, 2009.

[[Image:Compcog09-leon.jpg|center|300px]]

* Introduction seminar to Nao programming (Instructor [http://www.robotica-urjc.es Dr. Francisco Martín Rico]),

[[Image:Workshop-paco09.jpg|center|300px]]

=In the Press=

* [http://www.diariodevalladolid.es/noticias/valladolid/villanueva-augura-empresas-innoven-creceran-triple-proximos-cinco-anos_14514.html Award ''Innovadores Castilla y Leon 2015''] (El Mundo, 2015)

*[http://www.elmundo.es/economia/2014/12/31/54a2ee5b22601d42688b4580.html?intcmp=ULNOH002 Robots que medican y chalan], coverage by national newspaper of the group's activities (El Mundo, Dec, 2014)

* [http://www.diariodeleon.es/noticias/leon/premio-tic_845853.html Innova Award] to the best 2014 IT project, given by local newspaper (Diario de León).

* Article about Fernando's work on mobile robots [http://www.fernando.casadogarcia.es/innova.jpg Suplemento Innova (Diario de León)]

[[Image:DL-FernandoCasado.jpg|center|300px]]

* Juan Felipe was awarded the [http://www.diariodeleon.es/noticias/leon/juan-felipe-garcia_694743.html Young Talent 2012] title by Diario de León.
* Article about using mobile robots in [http://www.la-cronica.net/2011/02/28/leon/robots-moviles-para-asistir-a-personas-dependientes-113536.htm La Crónica].

* Conference on social robotics applied to handicapped people. Organized by CRE:

# Information about the event at [http://www.imserso.es/cresanandres_01/auxiliares/actualidad/novedades/2011/enero/IM_042486?dDocName=IM_042486 CRE's webpage].

# Press note at [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577647 Diario de León].

* Press Note about [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577402 Juan's PhD Viva].

[[Image:FotoDiarioLeon2011.jpg|center|300px]]

* Yet another info in the local newspaper, about a project on [http://www.diariodeleon.es/noticias/noticia.asp?pkid=571538 robotics for security].

* The same one in the local TV:

<videoflash>Z-lNsL-o8UY</videoflash>

* Another review in the local newspapers: [http://www.leonoticias.com/frontend/leonoticias/Dos-Robots-leoneses-Acudiran-A-La-Copa-De-Europa-De-Futb-vn45312-vst384 Leonnoticias], and [http://www.la-cronica.net/2010/03/11/leon/ingenieros-de-la-ule-estaran-en-la-champions-de-futbol-de-robots-72130.htm La Crónica], about the team entering the [http://www.robocup-mediterranean-open.org/ Mediterranean Open] (Rome, March 18 to 20, 2010).

* Interview in the local newspaper (Diario de León): [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495839 first one] and [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495838 second one].

[[Image:FotoDiarioLeon.jpg|center|300px]]

= RoboCup =

* We were proud members of the Spanish Team ([https://twitter.com/spiteam_robot SPITeam]) in the RoboCup Standard Platform providing a goalkeeper. SPITeam was a joint effort of the [robotics group] of the Rey Juan Carlos University, Rovira y Virgili University and Universidad de León. Last Team Description paper can be found [http://www.gsyc.es/~fmartin/SPiTeamApplication.pdf here]. Our last competition entrance was in [http://www.robocup2011.org/en/content.asp?PID={41EB0A0E-7EAF-4365-98F7-1D951C9B8D88}&PageFn=Teams Instambul 2011]

* We were previously also proud members of the [http://www.teamchaos.es TeamChaos] RoboCup SPL team. We have contributed the goalie to the team for the [http://www.robocup2009.org/ RoboCup 2009] and [http://www.robocup-german-open.de/en German Open 2009].

[[Image:Teamchaos-08.JPG|center|300px]]

Home

2016-02-23T13:21:39Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the [https://www.google.com/maps/@42.6150329,-5.5635688,203m/data=!3m1!1e3 MIC building] (Módulo de Investigación en Cibernética - Cybernetics Research Module). If you are interested in our work, you are welcome to pay us a visit.

[[Image:Mic2015.jpg|center|border|300px]]

We also hang around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática - Engineering School), which is [https://www.google.com/maps/@42.6141916,-5.5611339,203m/data=!3m1!1e3 here].

==News==

(February, 2016) Francisco Sedano gave a brief introduction to 3D prototyping technologies that can be applied to a wide variety of engineering applications: [http://robotica.unileon.es/mediawiki/documents/escaneado_triptico.pdf Scanning Course], [http://robotica.unileon.es/mediawiki/documents/impresoras_3D.pdf 3D Printing Course]. The training gave to the students the knowledge to develop and use 3D scanning and 3D printing technologies.

[[Image:20160218_195229.jpg|center|border|300px]]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(December, 2015) Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 2015) We took part in [http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015], the second competition event in the EU-funded robotics project RoCKIn. It took place in Lisbon (Portugal), from 21 to 23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

This is the Team Description Paper of our "Watermelon Project" team for the RoCKIn@home Challenge in Lisbon.

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

We want to express our gratitude to [http://www.mountain.es/ Mountain] for sponsoring our group in the RoCKIn Challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

We have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

Activities

2016-02-22T12:45:34Z

Victorm:

=Last Events=

==Francisco Lera's PhD dissertation==

Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

==RoCKIn 2015 (Lisboa) ==

[http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015], the second competition event in the EU-funded robotics project RoCKIn, was held in Lisbon (Portugal), 21-23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

It was a joint event with the [https://web.fe.up.pt/~robot2015/ ROBOT'2015], the Second Iberian Robotics Conference where we also presented [[a paper]]

==RoCKIn Event 2014 (Toulouse)==

(main article of this event [[RoCKInChallengeToulouse | here]])

==RoCKIn Camp 2014 (Rome)==

(main article of this event [[RoCKIn2014 | here]])

The team was in Rome testing the MYRABot robot. There, we learned different topics about vision, grasping, and natural language understanding. We tested the MB+ (MYRABot+) robot, a modification of MYRABot. Other robots were introduced in this competition, such as REEM from PAL Robotics.

[[Image:MBPAL1.JPG|center|border|300px]]

==Hack For Good 2013==

Proposals for [http://hackforgood.net HackForGood] can be found [[proposals-hackforfood-2013 | here]].

=Seminars=

* European Robotics Week Conference [[http://www.eu-robotics.net/eurobotics-week/events-2013/low-cost-service-robots-ule-watermelon-project-for-rockinhome-challenge.html | Low Cost Robots for RoCKIn Challenge ]]November 27th, 2013

* Course [[ProgramacionRobots2012|Introduction to Mobile Robot Programming (4th edition)]]. October 22nd to 26th , 2012

* Conference ''"Augmented reality for elderly medication control using a mobile robot"'' at the Curso de Verano [http://catedratelefonica.unileon.es/2012/07/09/participacion-de-la-catedra-en-el-curso-de-verano-envejecimiento-activo-una-aproximacion-a-la-realidad-actual/ Envejecimiento Activo].

[[Image:Foto-Curso-Verano-2012.jpeg|center|300px]]

* Conference ''"The role of edutainment robotics in mechatronics research and teaching"'' by [http://www.ipb.pt/~jllima/ José Luis Lima] and José Gonçalves from the [http://www.ipb.pt Politechical Institute of Bragança], on November 4, 2011.

[[Image:CharlaIPB-2011.jpg|center|300px]]

* Course [[ProgramacionRobots2011 | Mobile Robot Programming, medium level (1st edition)]]. March 2011.

* Seminar on BICA programming in the SPITeam for the RoboCup2011. Instructors: Francisco Martín and Carlos E. Agüero from [http://www.robotica-urjc.es URJC Robotics Group].

<wikiflv width="300" height="250" logo="true">/videos/CursoBica.flv</wikiflv>

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (3rd edition)]]. December 9 to 23, 2010.

[[Image:16122012.JPG|center|300px]]

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (2nd edition)]]. March 8 to 12, 2010.

<wikiflv width="300" height="250" logo="true">/videos/videoSumo20100312.flv</wikiflv>

* Course [http://roboticaleon2009.googlepages.com/ Mobile Robot programming]. July 1 to 7, 2009.

[[Image:Carrera.jpg|center|300px]]

* [http://www.compcog.org/ CompCog] Workshop ESF on Held in the University of León, May 26 to 27, 2009.

[[Image:Compcog09-leon.jpg|center|300px]]

* Introduction seminar to Nao programming (Instructor [http://www.robotica-urjc.es Dr. Francisco Martín Rico]),

[[Image:Workshop-paco09.jpg|center|300px]]

=In the Press=

* [http://www.diariodevalladolid.es/noticias/valladolid/villanueva-augura-empresas-innoven-creceran-triple-proximos-cinco-anos_14514.html Award ''Innovadores Castilla y Leon 2015''] (El Mundo, 2015)

*[http://www.elmundo.es/economia/2014/12/31/54a2ee5b22601d42688b4580.html?intcmp=ULNOH002 Robots que medican y chalan], coverage by national newspaper of the group's activities (El Mundo, Dec, 2014)

* [http://www.diariodeleon.es/noticias/leon/premio-tic_845853.html Innova Award] to the best 2014 IT project, given by local newspaper (Diario de León).

* Article about Fernando's work on mobile robots [http://www.fernando.casadogarcia.es/innova.jpg Suplemento Innova (Diario de León)]

[[Image:DL-FernandoCasado.jpg|center|300px]]

* Juan Felipe was awarded the [http://www.diariodeleon.es/noticias/leon/juan-felipe-garcia_694743.html Young Talent 2012] title by Diario de León.
* Article about using mobile robots in [http://www.la-cronica.net/2011/02/28/leon/robots-moviles-para-asistir-a-personas-dependientes-113536.htm La Crónica].

* Conference on social robotics applied to handicapped people. Organized by CRE:

# Information about the event at [http://www.imserso.es/cresanandres_01/auxiliares/actualidad/novedades/2011/enero/IM_042486?dDocName=IM_042486 CRE's webpage].

# Press note at [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577647 Diario de León].

* Press Note about [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577402 Juan's PhD Viva].

[[Image:FotoDiarioLeon2011.jpg|center|300px]]

* Yet another info in the local newspaper, about a project on [http://www.diariodeleon.es/noticias/noticia.asp?pkid=571538 robotics for security].

* The same one in the local TV:

<videoflash>Z-lNsL-o8UY</videoflash>

* Another review in the local newspapers: [http://www.leonoticias.com/frontend/leonoticias/Dos-Robots-leoneses-Acudiran-A-La-Copa-De-Europa-De-Futb-vn45312-vst384 Leonnoticias], and [http://www.la-cronica.net/2010/03/11/leon/ingenieros-de-la-ule-estaran-en-la-champions-de-futbol-de-robots-72130.htm La Crónica], about the team entering the [http://www.robocup-mediterranean-open.org/ Mediterranean Open] (Rome, March 18 to 20, 2010).

* Interview in the local newspaper (Diario de León): [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495839 first one] and [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495838 second one].

[[Image:FotoDiarioLeon.jpg|center|300px]]

= RoboCup =

* We were proud members of the Spanish Team ([https://twitter.com/spiteam_robot SPITeam]) in the RoboCup Standard Platform providing a goalkeeper. SPITeam was a joint effort of the [robotics group] of the Rey Juan Carlos University, Rovira y Virgili University and Universidad de León. Last Team Description paper can be found [http://www.gsyc.es/~fmartin/SPiTeamApplication.pdf here]. Our last competition entrance was in [http://www.robocup2011.org/en/content.asp?PID={41EB0A0E-7EAF-4365-98F7-1D951C9B8D88}&PageFn=Teams Instambul 2011]

* We were previously also proud members of the [http://www.teamchaos.es TeamChaos] RoboCup SPL team. We have contributed the goalie to the team for the [http://www.robocup2009.org/ RoboCup 2009] and [http://www.robocup-german-open.de/en German Open 2009].

[[Image:Teamchaos-08.JPG|center|300px]]

Activities

2016-02-22T12:45:15Z

Victorm:

=Last Events=

==Francisco Lera's PhD dissertation==

Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

==RoCKIn 2015 (Lisboa) ==

[[http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015]], the second competition event in the EU-funded robotics project RoCKIn, was held in Lisbon (Portugal), 21-23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

It was a joint event with the [[https://web.fe.up.pt/~robot2015/ ROBOT'2015]], the Second Iberian Robotics Conference where we also presented [[a paper]]

==RoCKIn Event 2014 (Toulouse)==

(main article of this event [[RoCKInChallengeToulouse | here]])

==RoCKIn Camp 2014 (Rome)==

(main article of this event [[RoCKIn2014 | here]])

The team was in Rome testing the MYRABot robot. There, we learned different topics about vision, grasping, and natural language understanding. We tested the MB+ (MYRABot+) robot, a modification of MYRABot. Other robots were introduced in this competition, such as REEM from PAL Robotics.

[[Image:MBPAL1.JPG|center|border|300px]]

==Hack For Good 2013==

Proposals for [http://hackforgood.net HackForGood] can be found [[proposals-hackforfood-2013 | here]].

=Seminars=

* European Robotics Week Conference [[http://www.eu-robotics.net/eurobotics-week/events-2013/low-cost-service-robots-ule-watermelon-project-for-rockinhome-challenge.html | Low Cost Robots for RoCKIn Challenge ]]November 27th, 2013

* Course [[ProgramacionRobots2012|Introduction to Mobile Robot Programming (4th edition)]]. October 22nd to 26th , 2012

* Conference ''"Augmented reality for elderly medication control using a mobile robot"'' at the Curso de Verano [http://catedratelefonica.unileon.es/2012/07/09/participacion-de-la-catedra-en-el-curso-de-verano-envejecimiento-activo-una-aproximacion-a-la-realidad-actual/ Envejecimiento Activo].

[[Image:Foto-Curso-Verano-2012.jpeg|center|300px]]

* Conference ''"The role of edutainment robotics in mechatronics research and teaching"'' by [http://www.ipb.pt/~jllima/ José Luis Lima] and José Gonçalves from the [http://www.ipb.pt Politechical Institute of Bragança], on November 4, 2011.

[[Image:CharlaIPB-2011.jpg|center|300px]]

* Course [[ProgramacionRobots2011 | Mobile Robot Programming, medium level (1st edition)]]. March 2011.

* Seminar on BICA programming in the SPITeam for the RoboCup2011. Instructors: Francisco Martín and Carlos E. Agüero from [http://www.robotica-urjc.es URJC Robotics Group].

<wikiflv width="300" height="250" logo="true">/videos/CursoBica.flv</wikiflv>

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (3rd edition)]]. December 9 to 23, 2010.

[[Image:16122012.JPG|center|300px]]

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (2nd edition)]]. March 8 to 12, 2010.

<wikiflv width="300" height="250" logo="true">/videos/videoSumo20100312.flv</wikiflv>

* Course [http://roboticaleon2009.googlepages.com/ Mobile Robot programming]. July 1 to 7, 2009.

[[Image:Carrera.jpg|center|300px]]

* [http://www.compcog.org/ CompCog] Workshop ESF on Held in the University of León, May 26 to 27, 2009.

[[Image:Compcog09-leon.jpg|center|300px]]

* Introduction seminar to Nao programming (Instructor [http://www.robotica-urjc.es Dr. Francisco Martín Rico]),

[[Image:Workshop-paco09.jpg|center|300px]]

=In the Press=

* [http://www.diariodevalladolid.es/noticias/valladolid/villanueva-augura-empresas-innoven-creceran-triple-proximos-cinco-anos_14514.html Award ''Innovadores Castilla y Leon 2015''] (El Mundo, 2015)

*[http://www.elmundo.es/economia/2014/12/31/54a2ee5b22601d42688b4580.html?intcmp=ULNOH002 Robots que medican y chalan], coverage by national newspaper of the group's activities (El Mundo, Dec, 2014)

* [http://www.diariodeleon.es/noticias/leon/premio-tic_845853.html Innova Award] to the best 2014 IT project, given by local newspaper (Diario de León).

* Article about Fernando's work on mobile robots [http://www.fernando.casadogarcia.es/innova.jpg Suplemento Innova (Diario de León)]

[[Image:DL-FernandoCasado.jpg|center|300px]]

* Juan Felipe was awarded the [http://www.diariodeleon.es/noticias/leon/juan-felipe-garcia_694743.html Young Talent 2012] title by Diario de León.
* Article about using mobile robots in [http://www.la-cronica.net/2011/02/28/leon/robots-moviles-para-asistir-a-personas-dependientes-113536.htm La Crónica].

* Conference on social robotics applied to handicapped people. Organized by CRE:

# Information about the event at [http://www.imserso.es/cresanandres_01/auxiliares/actualidad/novedades/2011/enero/IM_042486?dDocName=IM_042486 CRE's webpage].

# Press note at [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577647 Diario de León].

* Press Note about [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577402 Juan's PhD Viva].

[[Image:FotoDiarioLeon2011.jpg|center|300px]]

* Yet another info in the local newspaper, about a project on [http://www.diariodeleon.es/noticias/noticia.asp?pkid=571538 robotics for security].

* The same one in the local TV:

<videoflash>Z-lNsL-o8UY</videoflash>

* Another review in the local newspapers: [http://www.leonoticias.com/frontend/leonoticias/Dos-Robots-leoneses-Acudiran-A-La-Copa-De-Europa-De-Futb-vn45312-vst384 Leonnoticias], and [http://www.la-cronica.net/2010/03/11/leon/ingenieros-de-la-ule-estaran-en-la-champions-de-futbol-de-robots-72130.htm La Crónica], about the team entering the [http://www.robocup-mediterranean-open.org/ Mediterranean Open] (Rome, March 18 to 20, 2010).

* Interview in the local newspaper (Diario de León): [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495839 first one] and [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495838 second one].

[[Image:FotoDiarioLeon.jpg|center|300px]]

= RoboCup =

* We were proud members of the Spanish Team ([https://twitter.com/spiteam_robot SPITeam]) in the RoboCup Standard Platform providing a goalkeeper. SPITeam was a joint effort of the [robotics group] of the Rey Juan Carlos University, Rovira y Virgili University and Universidad de León. Last Team Description paper can be found [http://www.gsyc.es/~fmartin/SPiTeamApplication.pdf here]. Our last competition entrance was in [http://www.robocup2011.org/en/content.asp?PID={41EB0A0E-7EAF-4365-98F7-1D951C9B8D88}&PageFn=Teams Instambul 2011]

* We were previously also proud members of the [http://www.teamchaos.es TeamChaos] RoboCup SPL team. We have contributed the goalie to the team for the [http://www.robocup2009.org/ RoboCup 2009] and [http://www.robocup-german-open.de/en German Open 2009].

[[Image:Teamchaos-08.JPG|center|300px]]

RoboCup

2016-02-22T12:43:07Z

Victorm:

__TOC__

This page is currently being developed

= Members =

* Integration (Team Leader): Francisco Javier Rodríguez Lera
* Navigation and Control: Francisco Martín Rico
* Manipulation: Fernando Casado García
* Navigation: Jesús Balsa Comerón
* Perception: Diego García Ordás
* Team Director: Vicente Matellán Olivera

= Robots =

== OrBiOne ==

OrBiOne is the robot that we will be using for 2016 competitons. This mobile manipulation is developed by Robotnik [[http://www.robotnik.eu/robotnik-introduces-rb-one-new-mobile-manipulator/ more info]])

<center><videoflash>pU4avjbgFMQ</videoflash></center>



= Github Repository=

* Individuals:
https://github.com/FranLera

https://github.com/fmrico

https://github.com/casadofer

* Group
https://github.com/Robotica-ule/MYRABot

= Papers =

* [http://robotica.unileon.es/mediawiki/documents/TDP_OrbioneTeam.pdf Team Description paper ]

* List of group publications can be found in the University of [[Publications |León's robotics group publications page]] and in the URJC's [http://gsyc.es/~fmartin/#publications publications] page.

Home

2016-02-22T12:41:19Z

Victorm:

Home

2016-02-22T12:40:10Z

Victorm:

Home

2016-02-22T12:38:02Z

Victorm:

Home

2016-01-19T19:04:16Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the [https://www.google.com/maps/@42.6150329,-5.5635688,203m/data=!3m1!1e3 MIC building] (Módulo de Investigación en Cibernética - Cybernetics Research Module). If you are interested in our work, you are welcome to pay us a visit.

[[Image:Mic2015.jpg|center|border|300px]]

We also hang around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática - Engineering School), which is [https://www.google.com/maps/@42.6141916,-5.5611339,203m/data=!3m1!1e3 here].

==News==

(December, 2015) Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the RoCKIn@home Challenge in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

Activities

2016-01-19T19:02:21Z

Victorm:

=Last Events=

==Francisco Lera's PhD dissertation==

Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

==RoCKIn 2015 (Lisboa) ==

[[http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015]], the second and major competition event in the EU-funded robotics project RoCKIn, was held in Lisbon, 21-23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

It was a joint event to the [[https://web.fe.up.pt/~robot2015/ ROBOT'2015]], the Second Iberian Robotics Conference where we also presented [[a paper]]

==RoCKIn Event 2014 (Toulouse)==

(main article of this event [[RoCKInChallengeToulouse | here]])

==RoCKIn Camp 2014 (Rome)==

(main article of this event [[RoCKIn2014 | here]])

The team was in Rome testing the MYRABot robot. There, we learned different topics about vision, grasping, and natural language understanding. We tested the MB+ (MYRABot+) robot, a modification of MYRABot. Other robots were introduced in this competition, such as REEM from PAL Robotics.

[[Image:MBPAL1.JPG|center|border|300px]]

==Hack For Good 2013==

Proposals for [http://hackforgood.net HackForGood] can be found [[proposals-hackforfood-2013 | here]].

=Seminars=

* European Robotics Week Conference [[http://www.eu-robotics.net/eurobotics-week/events-2013/low-cost-service-robots-ule-watermelon-project-for-rockinhome-challenge.html | Low Cost Robots for RoCKIn Challenge ]]November 27th, 2013

* Course [[ProgramacionRobots2012|Introduction to Mobile Robot Programming (4th edition)]]. October 22nd to 26th , 2012

* Conference ''"Augmented reality for elderly medication control using a mobile robot"'' at the Curso de Verano [http://catedratelefonica.unileon.es/2012/07/09/participacion-de-la-catedra-en-el-curso-de-verano-envejecimiento-activo-una-aproximacion-a-la-realidad-actual/ Envejecimiento Activo].

[[Image:Foto-Curso-Verano-2012.jpeg|center|300px]]

* Conference ''"The role of edutainment robotics in mechatronics research and teaching"'' by [http://www.ipb.pt/~jllima/ José Luis Lima] and José Gonçalves from the [http://www.ipb.pt Politechical Institute of Bragança], on November 4, 2011.

[[Image:CharlaIPB-2011.jpg|center|300px]]

* Course [[ProgramacionRobots2011 | Mobile Robot Programming, medium level (1st edition)]]. March 2011.

* Seminar on BICA programming in the SPITeam for the RoboCup2011. Instructors: Francisco Martín and Carlos E. Agüero from [http://www.robotica-urjc.es URJC Robotics Group].

<wikiflv width="300" height="250" logo="true">/videos/CursoBica.flv</wikiflv>

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (3rd edition)]]. December 9 to 23, 2010.

[[Image:16122012.JPG|center|300px]]

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (2nd edition)]]. March 8 to 12, 2010.

<wikiflv width="300" height="250" logo="true">/videos/videoSumo20100312.flv</wikiflv>

* Course [http://roboticaleon2009.googlepages.com/ Mobile Robot programming]. July 1 to 7, 2009.

[[Image:Carrera.jpg|center|300px]]

* [http://www.compcog.org/ CompCog] Workshop ESF on Held in the University of León, May 26 to 27, 2009.

[[Image:Compcog09-leon.jpg|center|300px]]

* Introduction seminar to Nao programming (Instructor [http://www.robotica-urjc.es Dr. Francisco Martín Rico]),

[[Image:Workshop-paco09.jpg|center|300px]]

=In the Press=

* [http://www.diariodevalladolid.es/noticias/valladolid/villanueva-augura-empresas-innoven-creceran-triple-proximos-cinco-anos_14514.html Award ''Innovadores Castilla y Leon 2015''] (El Mundo, 2015)

*[http://www.elmundo.es/economia/2014/12/31/54a2ee5b22601d42688b4580.html?intcmp=ULNOH002 Robots que medican y chalan], coverage by national newspaper of the group's activities (El Mundo, Dec, 2014)

* [http://www.diariodeleon.es/noticias/leon/premio-tic_845853.html Innova Award] to the best 2014 IT project, given by local newspaper (Diario de León).

* Article about Fernando's work on mobile robots [http://www.fernando.casadogarcia.es/innova.jpg Suplemento Innova (Diario de León)]

[[Image:DL-FernandoCasado.jpg|center|300px]]

* Juan Felipe was awarded the [http://www.diariodeleon.es/noticias/leon/juan-felipe-garcia_694743.html Young Talent 2012] title by Diario de León.
* Article about using mobile robots in [http://www.la-cronica.net/2011/02/28/leon/robots-moviles-para-asistir-a-personas-dependientes-113536.htm La Crónica].

* Conference on social robotics applied to handicapped people. Organized by CRE:

# Information about the event at [http://www.imserso.es/cresanandres_01/auxiliares/actualidad/novedades/2011/enero/IM_042486?dDocName=IM_042486 CRE's webpage].

# Press note at [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577647 Diario de León].

* Press Note about [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577402 Juan's PhD Viva].

[[Image:FotoDiarioLeon2011.jpg|center|300px]]

* Yet another info in the local newspaper, about a project on [http://www.diariodeleon.es/noticias/noticia.asp?pkid=571538 robotics for security].

* The same one in the local TV:

<videoflash>Z-lNsL-o8UY</videoflash>

* Another review in the local newspapers: [http://www.leonoticias.com/frontend/leonoticias/Dos-Robots-leoneses-Acudiran-A-La-Copa-De-Europa-De-Futb-vn45312-vst384 Leonnoticias], and [http://www.la-cronica.net/2010/03/11/leon/ingenieros-de-la-ule-estaran-en-la-champions-de-futbol-de-robots-72130.htm La Crónica], about the team entering the [http://www.robocup-mediterranean-open.org/ Mediterranean Open] (Rome, March 18 to 20, 2010).

* Interview in the local newspaper (Diario de León): [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495839 first one] and [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495838 second one].

[[Image:FotoDiarioLeon.jpg|center|300px]]

= RoboCup =

* We were proud members of the Spanish Team ([https://twitter.com/spiteam_robot SPITeam]) in the RoboCup Standard Platform providing a goalkeeper. SPITeam was a joint effort of the [robotics group] of the Rey Juan Carlos University, Rovira y Virgili University and Universidad de León. Last Team Description paper can be found [http://www.gsyc.es/~fmartin/SPiTeamApplication.pdf here]. Our last competition entrance was in [http://www.robocup2011.org/en/content.asp?PID={41EB0A0E-7EAF-4365-98F7-1D951C9B8D88}&PageFn=Teams Instambul 2011]

* We were previously also proud members of the [http://www.teamchaos.es TeamChaos] RoboCup SPL team. We have contributed the goalie to the team for the [http://www.robocup2009.org/ RoboCup 2009] and [http://www.robocup-german-open.de/en German Open 2009].

[[Image:Teamchaos-08.JPG|center|300px]]

Activities

2016-01-19T19:01:34Z

Victorm:

=Last Events=

==Francisco Lera's PhD dissertation==

Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

==RoCKIn 2015 (Lisboa) ==

[[http://rockinrobotchallenge.eu/rockin2015.php RoCKIn2015]], the second and major competition event in the EU-funded robotics project RoCKIn, was held in Lisbon from 21-23 November 2015.

[[Image:RoCKIn2015.jpg|center|border|300px]]

It was a joint event to the [[https://web.fe.up.pt/~robot2015/ ROBOT'2015]], the Second Iberian Robotics Conference where we also presented [[a paper]]

==RoCKIn Event 2014 (Toulouse)==

(main article of this event [[RoCKInChallengeToulouse | here]])

==RoCKIn Camp 2014 (Rome)==

(main article of this event [[RoCKIn2014 | here]])

The team was in Rome testing the MYRABot robot. There, we learned different topics about vision, grasping, and natural language understanding. We tested the MB+ (MYRABot+) robot, a modification of MYRABot. Other robots were introduced in this competition, such as REEM from PAL Robotics.

[[Image:MBPAL1.JPG|center|border|300px]]

==Hack For Good 2013==

Proposals for [http://hackforgood.net HackForGood] can be found [[proposals-hackforfood-2013 | here]].

=Seminars=

* European Robotics Week Conference [[http://www.eu-robotics.net/eurobotics-week/events-2013/low-cost-service-robots-ule-watermelon-project-for-rockinhome-challenge.html | Low Cost Robots for RoCKIn Challenge ]]November 27th, 2013

* Course [[ProgramacionRobots2012|Introduction to Mobile Robot Programming (4th edition)]]. October 22nd to 26th , 2012

* Conference ''"Augmented reality for elderly medication control using a mobile robot"'' at the Curso de Verano [http://catedratelefonica.unileon.es/2012/07/09/participacion-de-la-catedra-en-el-curso-de-verano-envejecimiento-activo-una-aproximacion-a-la-realidad-actual/ Envejecimiento Activo].

[[Image:Foto-Curso-Verano-2012.jpeg|center|300px]]

* Conference ''"The role of edutainment robotics in mechatronics research and teaching"'' by [http://www.ipb.pt/~jllima/ José Luis Lima] and José Gonçalves from the [http://www.ipb.pt Politechical Institute of Bragança], on November 4, 2011.

[[Image:CharlaIPB-2011.jpg|center|300px]]

* Course [[ProgramacionRobots2011 | Mobile Robot Programming, medium level (1st edition)]]. March 2011.

* Seminar on BICA programming in the SPITeam for the RoboCup2011. Instructors: Francisco Martín and Carlos E. Agüero from [http://www.robotica-urjc.es URJC Robotics Group].

<wikiflv width="300" height="250" logo="true">/videos/CursoBica.flv</wikiflv>

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (3rd edition)]]. December 9 to 23, 2010.

[[Image:16122012.JPG|center|300px]]

* Course [[ProgramacionRobots2010|Introduction to Mobile Robot Programming (2nd edition)]]. March 8 to 12, 2010.

<wikiflv width="300" height="250" logo="true">/videos/videoSumo20100312.flv</wikiflv>

* Course [http://roboticaleon2009.googlepages.com/ Mobile Robot programming]. July 1 to 7, 2009.

[[Image:Carrera.jpg|center|300px]]

* [http://www.compcog.org/ CompCog] Workshop ESF on Held in the University of León, May 26 to 27, 2009.

[[Image:Compcog09-leon.jpg|center|300px]]

* Introduction seminar to Nao programming (Instructor [http://www.robotica-urjc.es Dr. Francisco Martín Rico]),

[[Image:Workshop-paco09.jpg|center|300px]]

=In the Press=

* [http://www.diariodevalladolid.es/noticias/valladolid/villanueva-augura-empresas-innoven-creceran-triple-proximos-cinco-anos_14514.html Award ''Innovadores Castilla y Leon 2015''] (El Mundo, 2015)

*[http://www.elmundo.es/economia/2014/12/31/54a2ee5b22601d42688b4580.html?intcmp=ULNOH002 Robots que medican y chalan], coverage by national newspaper of the group's activities (El Mundo, Dec, 2014)

* [http://www.diariodeleon.es/noticias/leon/premio-tic_845853.html Innova Award] to the best 2014 IT project, given by local newspaper (Diario de León).

* Article about Fernando's work on mobile robots [http://www.fernando.casadogarcia.es/innova.jpg Suplemento Innova (Diario de León)]

[[Image:DL-FernandoCasado.jpg|center|300px]]

* Juan Felipe was awarded the [http://www.diariodeleon.es/noticias/leon/juan-felipe-garcia_694743.html Young Talent 2012] title by Diario de León.
* Article about using mobile robots in [http://www.la-cronica.net/2011/02/28/leon/robots-moviles-para-asistir-a-personas-dependientes-113536.htm La Crónica].

* Conference on social robotics applied to handicapped people. Organized by CRE:

# Information about the event at [http://www.imserso.es/cresanandres_01/auxiliares/actualidad/novedades/2011/enero/IM_042486?dDocName=IM_042486 CRE's webpage].

# Press note at [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577647 Diario de León].

* Press Note about [http://www.diariodeleon.es/noticias/noticia.asp?pkid=577402 Juan's PhD Viva].

[[Image:FotoDiarioLeon2011.jpg|center|300px]]

* Yet another info in the local newspaper, about a project on [http://www.diariodeleon.es/noticias/noticia.asp?pkid=571538 robotics for security].

* The same one in the local TV:

<videoflash>Z-lNsL-o8UY</videoflash>

* Another review in the local newspapers: [http://www.leonoticias.com/frontend/leonoticias/Dos-Robots-leoneses-Acudiran-A-La-Copa-De-Europa-De-Futb-vn45312-vst384 Leonnoticias], and [http://www.la-cronica.net/2010/03/11/leon/ingenieros-de-la-ule-estaran-en-la-champions-de-futbol-de-robots-72130.htm La Crónica], about the team entering the [http://www.robocup-mediterranean-open.org/ Mediterranean Open] (Rome, March 18 to 20, 2010).

* Interview in the local newspaper (Diario de León): [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495839 first one] and [http://www.diariodeleon.es/noticias/noticia.asp?pkid=495838 second one].

[[Image:FotoDiarioLeon.jpg|center|300px]]

= RoboCup =

* We were proud members of the Spanish Team ([https://twitter.com/spiteam_robot SPITeam]) in the RoboCup Standard Platform providing a goalkeeper. SPITeam was a joint effort of the [robotics group] of the Rey Juan Carlos University, Rovira y Virgili University and Universidad de León. Last Team Description paper can be found [http://www.gsyc.es/~fmartin/SPiTeamApplication.pdf here]. Our last competition entrance was in [http://www.robocup2011.org/en/content.asp?PID={41EB0A0E-7EAF-4365-98F7-1D951C9B8D88}&PageFn=Teams Instambul 2011]

* We were previously also proud members of the [http://www.teamchaos.es TeamChaos] RoboCup SPL team. We have contributed the goalie to the team for the [http://www.robocup2009.org/ RoboCup 2009] and [http://www.robocup-german-open.de/en German Open 2009].

[[Image:Teamchaos-08.JPG|center|300px]]

RoboCup

2016-01-19T19:00:23Z

Victorm:

This page is currently being developed

= Members =

* Francisco Javier Rodríguez Lera
* Francisco Martín Rico
* Vicente Matellán Olivera
* Fernando Casado García
* Jesús Balsa Comerón
* Diego García Ordás

= Robots =

* OrBiOne

* Cervantes

= Papers =

Intention to participate

Description paper

People

2016-01-19T18:58:26Z

Victorm:

=Current members=

The robotics group of the University of León is currently made up by:

'''Faculty'''
# [http://robotica.unileon.es/~vmo Vicente Matellán Olivera] (head of the group)
# [https://www.linkedin.com/pub/camino-fern%C3%A1ndez-llamas/2b/169/74 Camino Fernández Llamas]
# Francisco Jesús Rodríguez Sedano

'''Postdocs'''
# [[Miguel-A-Conde | Miguel Ángel Conde González]]
# [[Francisco-phd|Francisco Javier Rodríguez Lera]] (Augmented Reality in Assistance Environments)
# [https://www.linkedin.com/pub/juan-felipe-garc%C3%ADa-sierra/65/44a/715?trk=pub-pbmap Juan Felipe García Sierra]

'''PhD students'''
# Gonzalo Esteban Costales (Haptic interaction) (October 2011 - )
# [[User:Victorm | Víctor Rodríguez]] ([[PhD-3D-Object-Tracking|3D object recognition and tracking]]) (October 2011 - )
# [[User:Fernando | Fernando Casado García]] ([[Mobile manipulation]]) (September 2013 - )
# Diego García Ordás (3D object recognition and tracking) (January 2016 - )

'''Undergrad students'''
# Jesús Balsa Comerón
# Clauda Álvarez Aparicio
# Pablo Blanco Medina
# Laura Inyesto Alonso

=Past members=

==Visitors==
# [https://www.linkedin.com/pub/t%C3%A2nia-carrera/72/164/1a3 Tânia Carrera] from Bragança, Portugal ([[ Tania-TFM-ROS04|Kinect with ROS and PCL]]) (February - June 2013)
# [http://souzamarcelo.weebly.com Marcelo de Souza] from Santa Catarina State University, [http://www.udesc.br UDESC]/CEAVI, Brasil ([[Marcelo-TurtleBot|Navigation Algorithms for Turtlebot]]) (October 2012 - February 2013)
# [http://homepage.univie.ac.at/andrea.ravignani/ Andrea Ravignani] from the University of Wien, Austria ([[Comparative-Cognition|Comparative Cognition]]) (October - December 2012)
# [http://neithan.weebly.com/ Jesús Martínez Gómez] from the University of Castilla la Mancha (2008 - 2009)

==2014-2015==

* Summer research residence
# Rubén Rodríguez Fernández ([[Ruben-RV-ROS01|Turtlebot Navigation]]) (July 2013 - )

==2013-2014==

* Master students
# [https://www.linkedin.com/pub/aitana-alonso-nogueira/5b/174/572?trk=pub-pbmap Aitana Alonso Nogueira] ([[Aitana-TFM-kinect-ROS05|Human tracking and recognition]]) ([[Aitana-TFM-Android-MYRA|Evolución de la Solución de Asistencia MYRA para su Despliegue en Plataformas Android]]) (September 2013 - September 2014)
* Summer research residence (undergrad)
# [https://es.linkedin.com/pub/jose-%C3%A1lvaro-fern%C3%A1ndez-%C3%A1lvarez/65/a00/344 Jose Álvaro Fernández Álvarez] ([[Alvaro-RV-HAPTICROS01|Haptics and ROS]]) (July - September 2013)

==2012-2013==
* PhD students
# [https://es.linkedin.com/pub/fernando-garc%C3%ADa-d%C3%ADaz-calvo/3b/b74/b16 Fernando García Diaz-Calvo] (Autonomous behavior generation in mobile robots) (October 2012 - discontinued)
* Master students
# Carlos Rodríguez Hernández ([[Carlos-TFM-MYRABot01|MYRA Robot: Hardware Update]]) (January - September 2013)
* Undergrad students
# [https://es.linkedin.com/pub/david-llamas-abad/88/548/9b6 David Llamas Abad] ([[David-PFC-ROS3|AR in ROS]])
# Julia Pérez del Río ([[JuliaP-PFC-Modular01|Modular robotics]])

==2011-2012==
* Master students
# Isidoro Gayo Vélez ([[Isidoro-TFM-Rovio02|Using rovio for tele-surveillance]]) (January 2011 - discontinued)
# [https://www.linkedin.com/pub/alvaro-botas-mu%C3%B1oz/3a/517/90 Álvaro Botas Muñoz] ([[AlvaroB-TFM-ROS01 |AR Drone]]) (November 2011 - July 2012)
# [[User:Fernando | Fernando Casado García]] ([[Fernando-TFM-ROS02|Controlling Turtlebot with ROS]]) (January 2012 - July 2012)
* Undergrad students
# Jesús Balsa Comerón ([[JesusB-PFC-Nao01|Nao Control]]) (March 2011 - discontinued)

==2010-2011==
* PhD students
# [https://www.linkedin.com/pub/juan-felipe-garc%C3%ADa-sierra/65/44a/715?trk=pub-pbmap Juan Felipe García Sierra] ([https://buleria.unileon.es/handle/10612/815 Aportaciones a la computación de atención visual y aplicación al control de un robot humanoide])
* Master students
# Pablo Lobato Criado ([[PabloL-Rovio04|Navigation with ROS and Rovio]])
# [https://es.linkedin.com/pub/v%C3%ADctor-manuel-gonz%C3%A1lez-mateos/38/356/827 Víctor Manuel González Mateos] ([[VictorM-TFM-Rovio03|Navigation with Rovio]])
# Julio San Román Gutiérrez (Augmented reality on turtlebot) (March 2010 - discontinued)
# [[User:Victorm | Víctor Rodríguez]] ([[TFM-Nao-Goalkeeper|Implementing a RoboCup goalie with Nao using BICA]]) (October 2010 - July 2011)
* Undergrad students
# [https://es.linkedin.com/pub/elena-ortega-fern%C3%A1ndez/35/aa/b17 Elena Ortega Fernández] ([[ElenaO-RV-proy1|NaoQI and BICA]]) (November 2010 - September 2011)
# [https://www.linkedin.com/pub/alvaro-botas-mu%C3%B1oz/3a/517/90 Álvaro Botas Muñoz] ([[AlvaroB-PFC-Drone02|AR Drone]]) (November 2010 - September 2011)
* Other
# [https://www.linkedin.com/pub/mario-castro-de-lera/14/886/3b9 Mario Castro de Lera] from the Robotics course of the Master in Cybernetics ([[MarioCL-Drone01|AR Drone]])
# [https://co.linkedin.com/pub/angel-rojo-roman/45/54a/0 Ángel Rojo Román] from the subject of Sistemas informáticos (Computer Systems) ([[AngelR-PFC-Drone01|AR Drone]]) (November 2010 - September 2011)

==2009-2010==
* Master students
# Julio San Román Gutiérrez ([[JulioSR-TFM-Rovio01|Augmented reality for teleoperation of a Rovio robot]])
# [https://es.linkedin.com/in/vfidalgo Víctor Fidalgo Villar] ([[VictorFV-TFM-Cenital|Ball tracking with overhead camera]])
* Undergrad students
# [https://es.linkedin.com/pub/pablo-flecha-guti%C3%A9rrez/24/907/973 Pablo Flecha Gutiérrez]
# [https://es.linkedin.com/pub/david-mart%C3%ADnez-mart%C3%ADnez/61/656/176 David Martínez Martínez] ([[DMartinez-PFC-Nao|Teleoperation of Nao robot]])
# [https://es.linkedin.com/in/hectorquintian Héctor Quintián Pardo] and Alba Rodríguez from Artificial Intelligence, Industrial Engineering (2nd cycle) ([[HAQR-IA-Rovio|Teleoperation of Rovio robot]]) (2009 - 2010, 2nd semester)
* Summer research residence (undergrad)
# [https://es.linkedin.com/pub/elena-ortega-fern%C3%A1ndez/35/aa/b17 Elena Ortega Fernández] ([[ElenaO-RV-proy1|NaoQI and BICA]]) (July - September 2010)
# [https://es.linkedin.com/in/hectorquintian Héctor Quintián Pardo] ([[HectorQ-RV-proy2|Teleoperation of Rovio robot]]) (July - September 2010)

Home

2016-01-19T18:53:30Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the [https://www.google.com/maps/@42.6150329,-5.5635688,203m/data=!3m1!1e3 MIC building] (Módulo de Investigación en Cibernética - Cybernetics Research Module). If you are interested in our work, you are welcome to pay us a visit.

[[Image:Mic2015.jpg|center|border|300px]]

We also hang around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática - Engineering School), which is [https://www.google.com/maps/@42.6141916,-5.5611339,203m/data=!3m1!1e3 here].

==News==

(December, 2015) Francisco J. Lera just became the newest doctor of the group. His defense was held on December the 11th and he got a "Sobresaliente Cum Laude". Congratulations, Fran!

[[Image:tesis-fran.jpg|center|border|300px]]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the RoCKIn@home Challenge in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

People

2016-01-14T07:49:35Z

Victorm:

=Current members=

The robotics group of the University of León is currently made up by:

'''Faculty'''
# [http://robotica.unileon.es/~vmo Vicente Matellán Olivera] (head of the group)
# [https://www.linkedin.com/pub/camino-fern%C3%A1ndez-llamas/2b/169/74 Camino Fernández Llamas]
# Francisco J. Rodríguez Sedano

'''Postdocs'''
# [[Miguel-A-Conde | Miguel Ángel Conde]]
# [[Francisco-phd|Francisco Javier Rodríguez Lera]] (Augmented Reality in Assistance Environments)
# [https://www.linkedin.com/pub/juan-felipe-garc%C3%ADa-sierra/65/44a/715?trk=pub-pbmap Juan Felipe García Sierra]

'''PhD students'''
# Gonzalo Esteban Costales (Haptic interaction) (October 2011 - )
# [[User:Victorm | Víctor Rodríguez]] ([[PhD-3D-Object-Tracking|3D object recognition and tracking]]) (October 2011 - )
# [[User:Fernando | Fernando Casado García]] ([[Mobile manipulation]]) (September 2013 - )

'''Undergrad students'''
# Jesús Balsa Comeron
# Clauda Álvarez Aparicio
# Pablo Blanco Medina
# Laura Inyesto Alonso

=Past members=

==Visitors==
# [https://www.linkedin.com/pub/t%C3%A2nia-carrera/72/164/1a3 Tânia Carrera] from Bragança, Portugal ([[ Tania-TFM-ROS04|Kinect with ROS and PCL]]) (February - June 2013)
# [http://souzamarcelo.weebly.com Marcelo de Souza] from Santa Catarina State University, [http://www.udesc.br UDESC]/CEAVI, Brasil ([[Marcelo-TurtleBot|Navigation Algorithms for Turtlebot]]) (October 2012 - February 2013)
# [http://homepage.univie.ac.at/andrea.ravignani/ Andrea Ravignani] from the University of Wien, Austria ([[Comparative-Cognition|Comparative Cognition]]) (October - December 2012)
# [http://neithan.weebly.com/ Jesús Martínez Gómez] from the University of Castilla la Mancha (2008 - 2009)

==2014-2015==

* Summer research residence
# Rubén Rodríguez Fernández ([[Ruben-RV-ROS01|Turtlebot Navigation]]) (July 2013 - )

==2013-2014==

* Master students
# [https://www.linkedin.com/pub/aitana-alonso-nogueira/5b/174/572?trk=pub-pbmap Aitana Alonso Nogueira] ([[Aitana-TFM-kinect-ROS05|Human tracking and recognition]]) ([[Aitana-TFM-Android-MYRA|Evolución de la Solución de Asistencia MYRA para su Despliegue en Plataformas Android]]) (September 2013 - September 2014)
* Summer research residence (undergrad)
# [https://es.linkedin.com/pub/jose-%C3%A1lvaro-fern%C3%A1ndez-%C3%A1lvarez/65/a00/344 Jose Álvaro Fernández Álvarez] ([[Alvaro-RV-HAPTICROS01|Haptics and ROS]]) (July - September 2013)

==2012-2013==
* PhD students
# [https://es.linkedin.com/pub/fernando-garc%C3%ADa-d%C3%ADaz-calvo/3b/b74/b16 Fernando García Diaz-Calvo] (Autonomous behavior generation in mobile robots) (October 2012 - discontinued)
* Master students
# Carlos Rodríguez Hernández ([[Carlos-TFM-MYRABot01|MYRA Robot: Hardware Update]]) (January - September 2013)
* Undergrad students
# [https://es.linkedin.com/pub/david-llamas-abad/88/548/9b6 David Llamas Abad] ([[David-PFC-ROS3|AR in ROS]])
# Julia Pérez del Río ([[JuliaP-PFC-Modular01|Modular robotics]])

==2011-2012==
* Master students
# Isidoro Gayo Vélez ([[Isidoro-TFM-Rovio02|Using rovio for tele-surveillance]]) (January 2011 - discontinued)
# [https://www.linkedin.com/pub/alvaro-botas-mu%C3%B1oz/3a/517/90 Álvaro Botas Muñoz] ([[AlvaroB-TFM-ROS01 |AR Drone]]) (November 2011 - July 2012)
# [[User:Fernando | Fernando Casado García]] ([[Fernando-TFM-ROS02|Controlling Turtlebot with ROS]]) (January 2012 - July 2012)
* Undergrad students
# Jesús Balsa Comerón ([[JesusB-PFC-Nao01|Nao Control]]) (March 2011 - discontinued)

==2010-2011==
* PhD students
# [https://www.linkedin.com/pub/juan-felipe-garc%C3%ADa-sierra/65/44a/715?trk=pub-pbmap Juan Felipe García Sierra] ([https://buleria.unileon.es/handle/10612/815 Aportaciones a la computación de atención visual y aplicación al control de un robot humanoide])
* Master students
# Pablo Lobato Criado ([[PabloL-Rovio04|Navigation with ROS and Rovio]])
# [https://es.linkedin.com/pub/v%C3%ADctor-manuel-gonz%C3%A1lez-mateos/38/356/827 Víctor Manuel González Mateos] ([[VictorM-TFM-Rovio03|Navigation with Rovio]])
# Julio San Román Gutiérrez (Augmented reality on turtlebot) (March 2010 - discontinued)
# [[User:Victorm | Víctor Rodríguez]] ([[TFM-Nao-Goalkeeper|Implementing a RoboCup goalie with Nao using BICA]]) (October 2010 - July 2011)
* Undergrad students
# [https://es.linkedin.com/pub/elena-ortega-fern%C3%A1ndez/35/aa/b17 Elena Ortega Fernández] ([[ElenaO-RV-proy1|NaoQI and BICA]]) (November 2010 - September 2011)
# [https://www.linkedin.com/pub/alvaro-botas-mu%C3%B1oz/3a/517/90 Álvaro Botas Muñoz] ([[AlvaroB-PFC-Drone02|AR Drone]]) (November 2010 - September 2011)
* Other
# [https://www.linkedin.com/pub/mario-castro-de-lera/14/886/3b9 Mario Castro de Lera] from the Robotics course of the Master in Cybernetics ([[MarioCL-Drone01|AR Drone]])
# [https://co.linkedin.com/pub/angel-rojo-roman/45/54a/0 Ángel Rojo Román] from the subject of Sistemas informáticos (Computer Systems) ([[AngelR-PFC-Drone01|AR Drone]]) (November 2010 - September 2011)

==2009-2010==
* Master students
# Julio San Román Gutiérrez ([[JulioSR-TFM-Rovio01|Augmented reality for teleoperation of a Rovio robot]])
# [https://es.linkedin.com/in/vfidalgo Víctor Fidalgo Villar] ([[VictorFV-TFM-Cenital|Ball tracking with overhead camera]])
* Undergrad students
# [https://es.linkedin.com/pub/pablo-flecha-guti%C3%A9rrez/24/907/973 Pablo Flecha Gutiérrez]
# [https://es.linkedin.com/pub/david-mart%C3%ADnez-mart%C3%ADnez/61/656/176 David Martínez Martínez] ([[DMartinez-PFC-Nao|Teleoperation of Nao robot]])
# [https://es.linkedin.com/in/hectorquintian Héctor Quintián Pardo] and Alba Rodríguez from Artificial Intelligence, Industrial Engineering (2nd cycle) ([[HAQR-IA-Rovio|Teleoperation of Rovio robot]]) (2009 - 2010, 2nd semester)
* Summer research residence (undergrad)
# [https://es.linkedin.com/pub/elena-ortega-fern%C3%A1ndez/35/aa/b17 Elena Ortega Fernández] ([[ElenaO-RV-proy1|NaoQI and BICA]]) (July - September 2010)
# [https://es.linkedin.com/in/hectorquintian Héctor Quintián Pardo] ([[HectorQ-RV-proy2|Teleoperation of Rovio robot]]) (July - September 2010)

Home

2016-01-14T07:48:36Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the [https://www.google.com/maps/@42.6150329,-5.5635688,203m/data=!3m1!1e3 MIC building] (Módulo de Investigación en Cibernética - Cybernetics Research Module). If you are interested in our work, you are welcome to pay us a visit.

[[Image:Mic2015.jpg|center|border|300px]]

We also hang around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática - Engineering School), which is [https://www.google.com/maps/@42.6141916,-5.5611339,203m/data=!3m1!1e3 here].

==News==

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the next RoCKIn@home Challenge. This competition will take place in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(February 6, 2015) We are happy to inform that we have been accepted into the [http://www.dis.uniroma1.it/~rockin/node/33 RoCKIn Camp 2015] that will take place on Peccioli (Pisa, Italy) next March!

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 15, 2014) On September 27th, we attended the [http://makerfaireleon.com/ Mini Maker Faire] event in León, as Makers.

[[Image:mmf_logo_leon.png|center|500px|link=http://makerfaireleon.com/]]

More information about the Maker Movement in [https://www.thegrommet.com/blog/the-maker-movement-infographic/ this infographic], and in the [http://makerfaire.com/ official web].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

Home

2016-01-14T07:43:52Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the [https://www.google.com/maps/@42.6150329,-5.5635688,203m/data=!3m1!1e3 MIC building] (Módulo de Investigación en Cibernética - Cybernetics Research Module), in the University of León:

[[Image:Mic2015.jpg|center|border|300px]]

You can also find us around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática - Engineering School). If you are interested in our work, you are welcome to pay us a visit.

==News==

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the next RoCKIn@home Challenge. This competition will take place in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(February 6, 2015) We are happy to inform that we have been accepted into the [http://www.dis.uniroma1.it/~rockin/node/33 RoCKIn Camp 2015] that will take place on Peccioli (Pisa, Italy) next March!

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 15, 2014) On September 27th, we attended the [http://makerfaireleon.com/ Mini Maker Faire] event in León, as Makers.

[[Image:mmf_logo_leon.png|center|500px|link=http://makerfaireleon.com/]]

More information about the Maker Movement in [https://www.thegrommet.com/blog/the-maker-movement-infographic/ this infographic], and in the [http://makerfaire.com/ official web].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

Home

2016-01-09T11:21:12Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the [https://www.google.com/maps/@42.6150329,-5.5635688,203m/data=!3m1!1e3 MIC building] (Módulo de Investigación en Cibernética - Cybernetics Research Module), in the University of León:

[[Image:Mic2015.jpg|center|border|300px]]

You can also find us around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática, Engineering School). If you are interested in our work, you are welcome to pay us a visit.

==News==

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the next RoCKIn@home Challenge. This competition will take place in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(February 6, 2015) We are happy to inform that we have been accepted into the [http://www.dis.uniroma1.it/~rockin/node/33 RoCKIn Camp 2015] that will take place on Peccioli (Pisa, Italy) next March!

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 15, 2014) On September 27th, we attended the [http://makerfaireleon.com/ Mini Maker Faire] event in León, as Makers.

[[Image:mmf_logo_leon.png|center|500px|link=http://makerfaireleon.com/]]

More information about the Maker Movement in [https://www.thegrommet.com/blog/the-maker-movement-infographic/ this infographic], and in the [http://makerfaire.com/ official web].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

Home

2016-01-08T22:42:23Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the MIC building (Módulo de Investigación en Cibernética - Cybernetics Research Module), in the University of León:

[[Image:Mic2015.jpg|center|border|300px]]

You can also find us around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática, Engineering School). If you are interested in our work, you are welcome to pay us a visit.

==News==

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the next RoCKIn@home Challenge. This competition will take place in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(February 6, 2015) We are happy to inform that we have been accepted into the [http://www.dis.uniroma1.it/~rockin/node/33 RoCKIn Camp 2015] that will take place on Peccioli (Pisa, Italy) next March!

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 15, 2014) On September 27th, we attended the [http://makerfaireleon.com/ Mini Maker Faire] event in León, as Makers.

[[Image:mmf_logo_leon.png|center|500px|link=http://makerfaireleon.com/]]

More information about the Maker Movement in [https://www.thegrommet.com/blog/the-maker-movement-infographic/ this infographic], and in the [http://makerfaire.com/ official web].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

Home

2016-01-08T22:37:20Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Foto-grupo2015.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

Our lab is in the second floor of the MIC building (Módulo de Investigación en Cibernética - Cybernetics Research Module), in the University of León:

[[Image:Mic2015.jpg|center|border|300px]]

You can also find us around the [http://centros.unileon.es/eiii/ EIII] (Escuela de Ingenierías Industrial e Informática, Engineering School). If you are interested in our work, you are welcome to pay us a visit.

==News==

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the next RoCKIn@home Challenge. This competition will take place in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(February 6, 2015) We are happy to inform that we have been accepted into the [http://www.dis.uniroma1.it/~rockin/node/33 RoCKIn Camp 2015] that will take place on Peccioli (Pisa, Italy) next March!

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 15, 2014) On September 27th, we attended the [http://makerfaireleon.com/ Mini Maker Faire] event in León, as Makers.

[[Image:mmf_logo_leon.png|center|500px|link=http://makerfaireleon.com/]]

More information about the Maker Movement in [https://www.thegrommet.com/blog/the-maker-movement-infographic/ this infographic], and in the [http://makerfaire.com/ official web].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

Home

2015-11-24T08:03:41Z

Victorm:

This is the official wiki of the Robotics Group of the [http://www.unileon.es University of León] (ULE). We are a group of researchers (faculty and students) interested in creating autonomous behaviour for mobile robots.

[[Image:Robotics group ULE.jpg|center|border|500px]]

We are proud members of [http://www.redaf.es RedAF] (Red Nacional de Investigación en Agentes Físicos), [http://retecog.net ReteCog] (Red Temática de Ciencia Cognitiva), and [http://www.euron.org/members/index#257 Euron].

==Location==

We are usually in the F6 lab of the Escuela de Ingenierías Industrial e Informática ([http://centros.unileon.es/eiii/ EIII]) in León (Spain). If you are interested in our work, you are welcome to pay us a visit.

==News==

(September 30, 2015) This is the Team Description Paper of our "Watermelon Project" team for the next RoCKIn@home Challenge. This competition will take place in Lisbon (Portugal).

[http://rockincompetition.eu/node/256 RoCKIn competition]

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(February 6, 2015) We are happy to inform that we have been accepted into the [http://www.dis.uniroma1.it/~rockin/node/33 RoCKIn Camp 2015] that will take place on Peccioli (Pisa, Italy) next March!

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(November 24, 2014) Coinciding with the [http://www.eu-robotics.net/eurobotics-week/about-eurobotics-week/ European Robotics Week], we are going to participate in the first competition event of RoCKIn. It will take place in the Cité de l'espace science museum in Toulouse, France.

You can follow up with us [[RoCKInChallengeToulouse|here]].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 19, 2014) CeRVaNTeS platform. Teleoperation using a [http://www.microsoft.com/hardware/es-es/p/xbox-360-wireless-controller-for-windows Xbox 360 wireless controller] ([[CeRVaNTeS' Teleoperation with xbox360 wireless controller (xbox360 controller+joy)|more info]]):

<center><videoflash>wF_gKr1KWfg#t=13</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(September 15, 2014) On September 27th, we attended the [http://makerfaireleon.com/ Mini Maker Faire] event in León, as Makers.

[[Image:mmf_logo_leon.png|center|500px|link=http://makerfaireleon.com/]]

More information about the Maker Movement in [https://www.thegrommet.com/blog/the-maker-movement-infographic/ this infographic], and in the [http://makerfaire.com/ official web].

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 18, 2014) CeRVaNTeS platform. Integration in [http://moveit.ros.org/ moveIt!] ([[Integration of CeRVaNTeS in moveIt! (gazebo+moveIt!)|more info]]):

<center><videoflash>bYtvNJHraVs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(July 11, 2014) [http://www.mountain.es/ Mountain] will sponsor our group for the next RoCKIn challenge.

[[Image:Lgo_mountain.png|center|330px|link=http://www.mountain.es/]]

For the last two months, we have been using Mountain hardware for research and development with our robots. The control software of the CeRVaNTeS platform runs on a [http://www.mountain.es/portatiles F-13] laptop. Soon, our MYRABot platorm will also mount one of their machines.

We want to express our sincere gratitude to [http://www.mountain.es/ Mountain] for their support to small research groups like ours.

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(June 16, 2014) [[Projects#Cervantes_Project|CeRVaNTeS platform]]. First steps and proof of concept:

<center><videoflash>JtDIRky7EKs</videoflash></center>

{| style="border: solid 0px white; width: 50%" align="center"
||
----
|}

(January 25-31, 2014) RoCKIn camp participation. Our team attended the [[RoCKIn2014|RoCKIn Camp 2014]] on Rome, Italy. There they tested the performance of the MYRABot system in a competition environment. Also, the members participated in the lectures and in the practical sessions about basic software infrastructure, perception, manipulation and human robot interaction through speech.

[[Image:MBPAL1.JPG|center|border|250px]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-18T19:47:11Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [https://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis (right hand rule).
float theta = 90.0f * (M_PI / 180.0f); // 90 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object. Another important note: the function <span style="color:#FF1493">"addCoordinateSystem()"</span> with those parameters seen above is marked as deprecated in new versions of PCL. From these on you will have to use something like this:

<syntaxhighlight lang=CPP>viewer.addCoordinateSystem(1.0, "reference", 0);</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [https://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [https://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [https://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [https://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [https://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [https://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://en.wikipedia.org/wiki/CUDA CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [https://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 4: 3D object recognition (descriptors)

2015-11-05T16:59:49Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

It is time to learn the basics of one of the most interesting applications of point cloud processing: 3D object recognition. Akin to 2D recognition, this technique relies on finding good keypoints (characteristic points) in the cloud, and matching them to a set of previously saved ones. But 3D has several advantages over 2D: namely, we will be able to estimate with decent accuracy the exact position and orientation of the object, relative to the sensor. Also, 3D object recognition tends to be more robust to clutter (crowded scenes where objects in the front occluding objects in the background). And finally, having information about the object's shape will help with collision avoidance or grasping operations.

In this first tutorial we will see what descriptors are, how many types are there available in PCL, and how to compute them.

=Overview=

The basis of 3D object recognition is to find a set of correspondences between two different clouds, one of them containing the object we are looking for. In order to do this, we need a way to compare points in an unambiguous manner. Until now, we have worked with points that store the XYZ coordinates, the RGB color... but none of those properties are unique enough. In two sequential scans, two points could share the same coordinates despite belonging to different surfaces, and using the color information takes us back to 2D recognition, will all the lightning related problems.

In a previous tutorial, we talked about [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Feature_estimation | features]], before introducing the normals. Normals are an example of feature, because they encode information about the vicinity of the point. That is, the neighboring points are taken into account when computing it, giving us an idea of how the surrounding surface is. But this is not enough. For a feature to be optimal, it must meet the following criteria:

# '''It must be robust to transformations''': rigid transformations (the ones that do not change the distance between points) like translations and rotations must not affect the feature. Even if we play with the cloud a bit beforehand, there should be no difference.
# '''It must be robust to noise''': measurement errors that cause noise should not change the feature estimation much.
# '''It must be resolution invariant''': if sampled with different density (like after performing downsampling), the result must be identical or similar.

This is where descriptors come in. They are more complex (and precise) signatures of a point, that encode a lot of information about the surrounding geometry. The purpose is to unequivocally identify a point across multiple point clouds, no matter the noise, resolution or transformations. Also, some of them capture additional data about the object they belong to, like the viewpoint (that lets us retrieve the pose).

[[Image:Correspondence.jpg|thumb|center|600px|Finding correspondences between point features of two clouds (image from http://pointclouds.org/).]]

There are many 3D descriptors implemented into PCL. Each one has its own method for computing unique values for a point. Some use the difference between the angles of the normals of the point and its neighbors, for example. Others use the distances between the points. Because of this, some are inherently better or worse for certain purposes. A given descriptor may be scale invariant, and another one may be better with occlusions and partial views of objects. Which one you choose depends on what you want to do.

After calculating the necessary values, an additional step is performed to reduce the descriptor size: the result is binned into an [https://en.wikipedia.org/wiki/Histogram histogram]. To do this, the value range of each variable that makes up the descriptor is divided into ''n'' subdivisions, and the number of occurrences in each one is counted. Try to imagine a descriptor that computes a single variable, that ranges from 1 to 100. We choose to create 10 bins for it, so the first bin would gather all occurrences between 1 and 10, the second from 11 to 20, and so on. We look at the value of the variable for the first point-neighbor pair, and it is 27, so we increment the value of the third bin by 1. We keep doing this until we get a final histogram for that keypoint. The bin size must be carefully chosen depending on how descriptive that variable is (the variables do not have to share the same number of bins, and also the bins do not have to be of the same size; if for example most values from the previous example fell in the 50-100 range then it would be sensible to have more bins of smaller size in that range).

Descriptors can be classified in two main categories: global and local. The process for computing and using each one (recognition pipeline) is different, so each will be explained in its own section in this article.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/how_features_work.php How 3D Features work in PCL]
** [https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features Overview and Comparison of Features]
** [http://www.pointclouds.org/assets/icra2013/pcl_features_icra13.pdf How does a good feature look like?]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

==Table==

The following table will give you a hint of how many descriptors are there in PCL, and some of their features:

{| class="wikitable sortable" style="margin: auto; text-align:center" cellpadding="15"
|+ 3D descriptors
|- style="vertical-align:middle;"
! scope="col"| Name
! scope="col"| Type
! scope="col"| Size<sup>†</sup>
! scope="col"| Custom PointType<sup>††</sup>
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#PFH|PFH]] (Point Feature Histogram)
| style="color: sienna;" | Local
| 125
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#FPFH|FPFH]] (Fast Point Feature Histogram)
| style="color: sienna;" | Local
| 33
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RSD|RSD]] (Radius-Based Surface Descriptor)
| style="color: sienna;" | Local
| 289
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#3DSC|3DSC]] (3D Shape Context)
| style="color: sienna;" | Local
| 1980
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#USC|USC]] (Unique Shape Context)
| style="color: sienna;" | Local
| 1960
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#SHOT|SHOT]] (Signatures of Histograms of Orientations)
| style="color: sienna;" | Local
| 352
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Spin_image|Spin image]]
| style="color: sienna;" | Local
| 153*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RIFT|RIFT]] (Rotation-Invariant Feature Transform)
| style="color: sienna;" | Local
| 32*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#NARF|NARF]] (Normal Aligned Radial Feature)
| style="color: sienna;" | Local
| 36
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RoPS|RoPS]] (Rotational Projection Statistics)
| style="color: sienna;" | Local
| 135*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#VFH|VFH]] (Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#CVFH|CVFH]] (Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#OUR-CVFH|OUR-CVFH]] (Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#ESF|ESF]] (Ensemble of Shape Functions)
| style="color: darkcyan;" | Global
| 640
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GFPFH|GFPFH]] (Global Fast Point Feature Histogram)
| style="color: darkcyan;" | Global
| 16
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GRSD|GRSD]] (Global Radius-Based Surface Descriptor)
| style="color: darkcyan;" | Global
| 21
| style="color: green;" | Yes
|}

† Values marked with an asterisk (*) indicate that the descriptor's size depends on some parameter(s), and the one given is for the default values.

†† Descriptors without their own custom PointType use the generic <span style="color:#FF1493">"pcl::Histogram<>"</span> type. See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|Saving and loading]].

Optionally, you can download a document with a less simple version of the table. Page format is A4, landscape:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Downloads''':
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.odt Table as original ODT] (uses font embedding, requires LibreOffice 4)
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.pdf Table as PDF]
</div>

=Local descriptors=

Local descriptors are computed for individual points that we give as input. They have no notion of what an object is, they just describe how the local geometry is around that point. Usually, it is your task to choose which points you want a descriptor to be computed for: the ''keypoints''. Most of the time, you can get away by just performing a downsampling and choosing all remaining points, but keypoint detectors are available, like the one used for [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF]], or [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#ISS|ISS]].

Local descriptors are used for object recognition and [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration | registration]]. Now we will see which ones are implemented into PCL.

==PFH==

PFH stands for Point Feature Histogram. It is one of the most important descriptors offered by PCL and the basis of others such as FPFH. The PFH tries to capture information of the geometry surrounding the point by analyzing the difference between the directions of the normals in the vicinity (and because of this, an imprecise normal estimation may produce low-quality descriptors).

First, the algorithm pairs all points in the vicinity (not just the chosen keypoint with its neighbors, but also the neighbors with themselves). Then, for each pair, a [https://en.wikipedia.org/wiki/Darboux_frame fixed coordinate frame] is computed from their normals. With this frame, the difference between the normals can be encoded with 3 angular variables. These variables, together with the euclidean distance between the points, are saved, and then binned to an histogram when all pairs have been computed. The final descriptor is the concatenation of the histograms of each variable (4 in total).

<center><gallery widths=300px>
File:PFH_neighbors.png | Point pairs established when computing the PFH for a point (image from http://pointclouds.org).
File:PFH_frame.png | Fixed coordinate frame and angular features computed for one of the pairs (image from http://pointclouds.org).
</gallery></center>

Computing descriptors in PCL is very easy, and the PFH is not an exception:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/pfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the PFH descriptors for each point.
pcl::PointCloud<pcl::PFHSignature125>::Ptr descriptors(new pcl::PointCloud<pcl::PFHSignature125>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// PFH estimation object.
pcl::PFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::PFHSignature125> pfh;
pfh.setInputCloud(cloud);
pfh.setInputNormals(normals);
pfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
pfh.setRadiusSearch(0.05);

pfh.compute(*descriptors);
}</syntaxhighlight>

As you can see, PCL uses the <span style="color:#FF1493">"PFHSignature125"</span> type to save the descriptor to. This means that the descriptor's size is 125 (the dimensionality of the feature vector). Dividing a feature in ''D'' dimensional space in ''B'' divisions requires a total of B<sup>D</sup> bins. The original proposal makes use of the distance between the points, but the implementation of PCL does not, as it was not considered discriminative enough (specially in 2.5D scans, where the distance between points increases the further away from the sensor). Hence, the remaining features (with one dimension each) can be divided in 5 divisions, resulting in a 125-dimensional (5<sup>3</sup>) vector.

The final object that stores the computed descriptors can be handled like a normal cloud (even saved to, or read from, disk), and it has the same number of "points" than the original one. The <span style="color:#FF1493">"PFHSignature125"</span> object at position 0, for example, stores the PFH descriptor for the <span style="color:#FF1493">"PointXYZ"</span> point at the same position in the cloud.

For additional details about the descriptor, check the original publications listed below, or PCL's tutorial.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': PFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pfh_estimation.php Point Feature Histograms (PFH) descriptors]
* '''Publications''':
** [http://ias.in.tum.de/_media/spezial/bib/rusu08ias.pdf Persistent Point Feature Histograms for 3D Point Clouds] (Radu Bogdan Rusu et al., 2008)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.4, page 50)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_p_f_h_estimation.html pcl::PFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_PFH.tar.gz Download]
</div>

===FPFH===

PFH gives accurate results, but it has a drawback: it is too computationally expensive to perform at real time. For a cloud of ''n'' keypoints with ''k'' neighbors considered, it has a [https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations complexity] of ''O(nk<sup>2</sup>)''. Because of this, a derived descriptor was created, named FPFH (Fast Point Feature Histogram).

The FPFH considers only the direct connections between the current keypoint and its neighbors, removing additional links between neighbors. This takes the complexity down to ''O(nk)''. Because of this, the resulting histogram is referred to as SPFH (Simplified Point Feature Histogram). The reference frame and the angular variables are computed as always.

[[Image:FPFH_neighbors.png|thumb|center|400px|Point pairs established when computing the FPFH for a point (image from http://pointclouds.org).]]

To account for the loss of these extra connections, an additional step takes place after all histograms have been computed: the SPFHs of a point's neighbors are "merged" with its own, weighted according to the distance. This has the effect of giving a point surface information of points as far away as 2 times the radius used. Finally, the 3 histograms (distance is not used) are concatenated to compose the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/fpfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the FPFH descriptors for each point.
pcl::PointCloud<pcl::FPFHSignature33>::Ptr descriptors(new pcl::PointCloud<pcl::FPFHSignature33>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// FPFH estimation object.
pcl::FPFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::FPFHSignature33> fpfh;
fpfh.setInputCloud(cloud);
fpfh.setInputNormals(normals);
fpfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
fpfh.setRadiusSearch(0.05);

fpfh.compute(*descriptors);
}</syntaxhighlight>

An additional implementation of the FPFH estimation that takes advantage of multithreaded optimizations (with the [https://en.wikipedia.org/wiki/OpenMP OpenMP API]) is available in the class <span style="color:#FF1493">"FPFHEstimationOMP"</span>. Its interface is identical to the standard unoptimized implementation. Using it will result in a big performance boost on multi-core systems, meaning faster computation times. Remember to include the header <span style="color:#FF1493">"fpfh_omp.h"</span> instead.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': FPFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/fpfh_estimation.php Fast Point Feature Histograms (FPFH) descriptors]
* '''Publications''':
** [http://files.rbrusu.com/publications/Rusu09ICRA.pdf Fast Point Feature Histograms (FPFH) for 3D Registration] (Radu Bogdan Rusu et al., 2009)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.5, page 57)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation.html pcl::FPFHEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation_o_m_p.html pcl::FPFHEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_FPFH.tar.gz Download]
</div>

==RSD==

The Radius-Based Surface Descriptor encodes the radial relationship of the point and its neighborhood. For every pair of the keypoint with a neighbor, the algorithm computes the distance between them, and the difference between their normals. Then, by assuming that both points lie on the surface of a sphere, said sphere is found by fitting not only the points, but also the normals (otherwise, there would be infinite possible spheres). Finally, from all the point-neighbor spheres, only the ones with the maximum and minimum radii are kept and saved to the descriptor of that point.

As you may have deduced already, when two points lie on a flat surface, the sphere radius will be infinite. If, on the other hand, they lie on the curved face of a cylinder, the radius will be more or less the same as that of the cylinder. This allows us to tell objects apart with RSD. The algorithm takes a parameter that sets the maximum radius at which the points will be considered to be part of a plane.

<center><gallery widths=300px>
File:RSD_sphere.png | Sphere that fits two points with their normals (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
File:RSD_values.png | RSD values for a cloud; points on a flat surface have maximum values (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
</gallery></center>

This is the code for compiling the RSD descriptor:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/rsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RSD descriptors for each point.
pcl::PointCloud<pcl::PrincipalRadiiRSD>::Ptr descriptors(new pcl::PointCloud<pcl::PrincipalRadiiRSD>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// RSD estimation object.
pcl::RSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::PrincipalRadiiRSD> rsd;
rsd.setInputCloud(cloud);
rsd.setInputNormals(normals);
rsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
rsd.setRadiusSearch(0.05);
// Plane radius. Any radius larger than this is considered infinite (a plane).
rsd.setPlaneRadius(0.1);
// Do we want to save the full distance-angle histograms?
rsd.setSaveHistograms(false);

rsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

Optionally, you can use the <span style="color:#FF1493">"setSaveHistograms()"</span> function to enable the saving of the full distance-angle histograms, and then use <span style="color:#FF1493">"getHistograms()"</span> to retrieve them.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Maximum Radius, [Subdivisions], [Save full histograms]
* '''Output''': RSD descriptors
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.391.7398&rep=rep1&type=pdf General 3D Modelling of Novel Objects from a Single View] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_s_d_estimation.html pcl::RSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RSD.tar.gz Download]
</div>

==3DSC==

The 3D Shape Context is a descriptor that extends its existing 2D counterpart to the third dimension. It works by creating a support structure (a sphere, to be precise) centered at the point we are computing the descriptor for, with the given search radius. The "north pole" of that sphere (the notion of "up") is pointed to match the normal at that point. Then, the sphere is divided in 3D regions or bins. In the first 2 coordinates (azimuth and elevation) the divisions are equally spaced, but in the third (the radial dimension), divisions are logarithmically spaced so they are smaller towards the center. A minimum radius can be specified to prevent very small bins, that would be too sensitive to small changes in the surface.

[[Image:3DSC_support_structure.png|thumb|center|200px|Support structure to compute the 3DSC for a point (image from [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf original paper]).]]

For each bin, a weighted count is accumulated for every neighboring point that lies within. The weight depends on the volume of the bin and the local point density (number of points around the current neighbor). This gives the descriptor some degree of resolution invariance.

We have mentioned that the sphere is given the direction of the normal. This still leaves one degree of freedom (only two axes have been locked, the azimuth remains free). Because of this, the descriptor so far does not cope with rotation. To overcome this (so the same point in two different clouds has the same value), the support sphere is rotated around the normal ''N'' times (a number of degrees that corresponds with the divisions in the azimuth) and the process is repeated for each, giving a total of ''N'' descriptors for that point.

You can compute the 3DSC descriptor the following way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/3dsc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the 3DSC descriptors for each point.
pcl::PointCloud<pcl::ShapeContext1980>::Ptr descriptors(new pcl::PointCloud<pcl::ShapeContext1980>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// 3DSC estimation object.
pcl::ShapeContext3DEstimation<pcl::PointXYZ, pcl::Normal, pcl::ShapeContext1980> sc3d;
sc3d.setInputCloud(cloud);
sc3d.setInputNormals(normals);
sc3d.setSearchMethod(kdtree);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
sc3d.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
sc3d.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
sc3d.setPointDensityRadius(0.05 / 5.0);

sc3d.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Minimal radius, Point density radius
* '''Output''': 3DSC descriptors
* '''Publication''':
** [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf Recognizing Objects in Range Data Using Regional Point Descriptors] (Andrea Frome et al., 2004)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_shape_context3_d_estimation.html pcl::ShapeContext3DEstimation]
* [http://robotica.unileon.es/~victorm/PCL_3DSC.tar.gz Download]
</div>

===USC===

The Unique Shape Context descriptor extends the 3DSC by defining a local reference frame, in order to provide an unique orientation for each point. This not only improves the accuracy of the descriptor, it also reduces its size, as computing multiple descriptors to account for orientation is no longer necessary.

You can check the second publication listed below to learn more about how the LRF is computed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/usc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the USC descriptors for each point.
pcl::PointCloud<pcl::UniqueShapeContext1960>::Ptr descriptors(new pcl::PointCloud<pcl::UniqueShapeContext1960>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// USC estimation object.
pcl::UniqueShapeContext<pcl::PointXYZ, pcl::UniqueShapeContext1960, pcl::ReferenceFrame> usc;
usc.setInputCloud(cloud);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
usc.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
usc.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
usc.setPointDensityRadius(0.05 / 5.0);
// Set the radius to compute the Local Reference Frame.
usc.setLocalRadius(0.05);

usc.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk). For 1.7 and below, change UniqueShapeContext1960 to ShapeContext1980, and edit CMakeLists.txt.'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Minimal radius, Point density radius, Local radius
* '''Output''': USC descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/3dor10.pdf Unique Shape Context for 3D Data Description] (Federico Tombari et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_unique_shape_context.html pcl::UniqueShapeContext]
* [http://robotica.unileon.es/~victorm/PCL_USC.tar.gz Download]
</div>

==SHOT==

SHOT stands for Signature of Histograms of Orientations. Like 3DSC, it encodes information about the topology (surface) withing a spherical support structure. This sphere is divided in 32 bins or volumes, with 8 divisions along the azimuth, 2 along the elevation, and 2 along the radius. For every volume, a one-dimensional local histogram is computed. The variable chosen is the angle between the normal of the keypoint and the current point within that volume (to be precise, the cosine, which was found to be better suitable).

[[Image:SHOT_support_structure.png|thumb|center|200px|Support structure to compute SHOT. Only 4 azimuth divisions are shown for clarity (image from [http://vision.deis.unibo.it/fede/papers/eccv10.pdf original paper]).]]

When all local histograms have been computed, they are stitched together in a final descriptor. Like the USC descriptor, SHOT makes use of a local reference frame, making it rotation invariant. It is also robust to noise and clutter.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the SHOT descriptors for each point.
pcl::PointCloud<pcl::SHOT352>::Ptr descriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// SHOT estimation object.
pcl::SHOTEstimation<pcl::PointXYZ, pcl::Normal, pcl::SHOT352> shot;
shot.setInputCloud(cloud);
shot.setInputNormals(normals);
// The radius that defines which of the keypoint's neighbors are described.
// If too large, there may be clutter, and if too small, not enough points may be found.
shot.setRadiusSearch(0.02);

shot.compute(*descriptors);
}</syntaxhighlight>

Like with FPFH, a multithreading-optimized variant is available with <span style="color:#FF1493">"SHOTEstimationOMP"</span>, that makes use of OpenMP. You need to include the header <span style="color:#FF1493">"shot_omp.h"</span>. Also, another variant that uses the texture for matching is available, <span style="color:#FF1493">"SHOTColorEstimation"</span>, with an optimized version too (see the second publication for more details). It outputs a <span style="color:#FF1493">"SHOT1344"</span> descriptor.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius
* '''Output''': SHOT descriptors
* '''Publications''':
** [http://vision.deis.unibo.it/fede/papers/eccv10.pdf Unique Signatures of Histograms for Local Surface Description] (Federico Tombari et al., 2010)
** [http://www.vision.deis.unibo.it/fede/papers/icip11.pdf A Combined Texture-Shaped Descriptor for Enhanced 3D Feature Matching] (Federico Tombari et al., 2011)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation.html pcl::SHOTEstimation],
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation.html pcl::SHOTColorEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation_o_m_p.html pcl::SHOTEstimationOMP]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation_o_m_p.html pcl::SHOTColorEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_SHOT.tar.gz Download]
</div>

==Spin image==

The Spin Image (SI) is the oldest descriptor we are going to see here. It has been around since 1997, but it still sees some use for certain applications. It was originally designed to describe surfaces made by vertices, edges and polygons, but it has been since adapted for point clouds. The descriptor is unlike all others in that the output resembles an image that can be compared with another with the usual means.

The support structure used is a cylinder, centered at the point, with a given radius and height, and aligned with the normal. This cylinder is divided radially and vertically into volumes. For each one, the number of neighbors lying inside is added up, eventually producing a descriptor. Weighting and interpolation are used to improve the result. The final descriptor can be seen as a grayscale image where dark areas correspond to volumes with higher point density.

[[Image:Spin images.png|thumb|center|400px|Spin images computed for 3 points of a model (image from [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf original thesis]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/spin_image.h>

// A handy typedef.
typedef pcl::Histogram<153> SpinImage;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the spin image for each point.
pcl::PointCloud<SpinImage>::Ptr descriptors(new pcl::PointCloud<SpinImage>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Spin image estimation object.
pcl::SpinImageEstimation<pcl::PointXYZ, pcl::Normal, SpinImage> si;
si.setInputCloud(cloud);
si.setInputNormals(normals);
// Radius of the support cylinder.
si.setRadiusSearch(0.02);
// Set the resolution of the spin image (the number of bins along one dimension).
// Note: you must change the output histogram size to reflect this.
si.setImageWidth(8);

si.compute(*descriptors);
}</syntaxhighlight>

The Spin Image estimation object provides more methods for tuning the estimation, so checking the API is recommended.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius, Image resolution
* '''Output''': Spin images
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf Spin-Images: A Representation for 3-D Surface Matching] (Andrew Edie Johnson, 1997)
** [http://www.cs.jhu.edu/~misha/Papers/Johnson99.pdf Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes] (Andrew Edie Johnson and Martial Hebert, 1999)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_spin_image_estimation.html pcl::SpinImageEstimation]
* [http://robotica.unileon.es/~victorm/PCL_spin_image.tar.gz Download]
</div>

==RIFT==

The Rotation-Invariant Feature Transform, like the spin image, takes some concepts from 2D features, in this case from the Scale-Invariant Feature Transform ([https://en.wikipedia.org/wiki/Scale-invariant_feature_transform SIFT]). It is the only descriptor seen here that requires intensity information in order to compute it (it can be obtained from the RGB color values). This means, of course, that you will not be able to use RIFT with standard XYZ clouds, you also need the texture.

In the first step, a circular patch (with the given radius) is fitted on the surface the point lies on. This patch is divided into concentric rings, according to the chosen distance bin size. Then, an histogram is populated with all the point's neighbors lying inside a sphere centered at that point and with the mentioned radius. The distance and the orientation of the intensity gradient at each point are considered. To make it rotation invariant, the angle between the gradient orientation and the vector pointing outward from the center of the patch is measured.

[[Image:RIFT.png|thumb|center|600px|RIFT feature values at 3 different locations in the descriptor (image from [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf original paper]).]]

The authors' original implementation uses 4 rings and 8 histogram orientations, which produce a descriptor of size 32. RIFT is not robust to texture flipping, though this was never considered a big issue.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/intensity_gradient.h>
#include <pcl/features/rift.h>

// A handy typedef.
typedef pcl::Histogram<32> RIFT32;

int
main(int argc, char** argv)
{
// Object for storing the point cloud with color information.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloudColor(new pcl::PointCloud<pcl::PointXYZRGB>);
// Object for storing the point cloud with intensity value.
pcl::PointCloud<pcl::PointXYZI>::Ptr cloudIntensity(new pcl::PointCloud<pcl::PointXYZI>);
// Object for storing the intensity gradients.
pcl::PointCloud<pcl::IntensityGradient>::Ptr gradients(new pcl::PointCloud<pcl::IntensityGradient>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RIFT descriptor for each point.
pcl::PointCloud<RIFT32>::Ptr descriptors(new pcl::PointCloud<RIFT32>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloudColor) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Convert the RGB to intensity.
pcl::PointCloudXYZRGBtoXYZI(*cloudColor, *cloudIntensity);

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZI, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudIntensity);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZI>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZI>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Compute the intensity gradients.
pcl::IntensityGradientEstimation < pcl::PointXYZI, pcl::Normal, pcl::IntensityGradient,
pcl::common::IntensityFieldAccessor<pcl::PointXYZI> > ge;
ge.setInputCloud(cloudIntensity);
ge.setInputNormals(normals);
ge.setRadiusSearch(0.03);
ge.compute(*gradients);

// RIFT estimation object.
pcl::RIFTEstimation<pcl::PointXYZI, pcl::IntensityGradient, RIFT32> rift;
rift.setInputCloud(cloudIntensity);
rift.setSearchMethod(kdtree);
// Set the intensity gradients to use.
rift.setInputGradient(gradients);
// Radius, to get all neighbors within.
rift.setRadiusSearch(0.02);
// Set the number of bins to use in the distance dimension.
rift.setNrDistanceBins(4);
// Set the number of bins to use in the gradient orientation dimension.
rift.setNrGradientBins(8);
// Note: you must change the output histogram size to reflect the previous values.

rift.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Intensity gradients, Radius, Distance bins, Gradient bins
* '''Output''': RIFT descriptors
* '''Publication''':
** [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf A Sparse Texture Representation Using Local Affine Regions] (Svetlana Lazebnik et al., 2005)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_r_i_f_t_estimation.html pcl::RIFTEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_intensity_gradient_estimation.html pcl::IntensityGradientEstimation]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_intensity_gradient.html pcl::IntensityGradient]
* [http://robotica.unileon.es/~victorm/PCL_RIFT.tar.gz Download]
</div>

==NARF==

The Normal Aligned Radial Feature is the only descriptor here that does not take a point cloud as input. Instead, it works with range images. A range image is a common RGB image in which the distance to the point that corresponds to a certain pixel is encoded as a color value in the [https://en.wikipedia.org/wiki/Visible_spectrum visible light spectrum]: the points that are closer to the camera would be violet, while the points near the maximum sensor range would be red.

NARF also requires us to find suitable keypoints to compute the descriptor for. NARF keypoints are located near an object's corners, and this also requires to find the borders (transitions from foreground to background), which are trivial to find with a range image. Because of this lengthy pipeline, I will describe the whole process in different sections.

===Obtaining a range image===

Because we always work with point clouds, I will now explain how you can convert one into a range image, in order to use it for the NARF descriptor. PCL provides a couple of handy classes to perform the conversion, given that you fill the camera data correctly.

A range image can be created in two ways. First, we can use spherical projection, which would give us an image similar to the ones produced by a LIDAR sensor. Second, we can use planar projection, which is better suitable for camera-like sensors as the Kinect or the Xtion, and will not have the characteristic distortion of the first one.

====Spherical projection====

The following code will take a point cloud and create a range image from it, using spherical projection:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the range image object:

// Angular resolution is the angular distance between pixels.
// Kinect: 57° horizontal FOV, 43° vertical FOV, 640x480 (chosen here).
// Xtion: 58° horizontal FOV, 45° vertical FOV, 640x480.
float angularResolutionX = (float)(57.0f / 640.0f * (M_PI / 180.0f));
float angularResolutionY = (float)(43.0f / 480.0f * (M_PI / 180.0f));
// Maximum horizontal and vertical angles. For example, for a full panoramic scan,
// the first would be 360º. Choosing values that adjust to the real sensor will
// decrease the time it takes, but don't worry. If the values are bigger than
// the real ones, the image will be automatically cropped to discard empty zones.
float maxAngleX = (float)(60.0f * (M_PI / 180.0f));
float maxAngleY = (float)(50.0f * (M_PI / 180.0f));
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;
// Border size. If greater than 0, a border of "unobserved" points will be left
// in the image when it is cropped.
int borderSize = 1;

// Range image object.
pcl::RangeImage rangeImage;
rangeImage.createFromPointCloud(*cloud, angularResolutionX, angularResolutionY,
maxAngleX, maxAngleY, sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange, borderSize);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Range image");
viewer.showRangeImage(rangeImage);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

Here you can see an example of the output range image:

[[Image:Range_image_spherical.png|thumb|center|400px|Range image of a point cloud, using spherical projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Angular resolution, Maximum angle, Sensor pose, Coordinate frame, Noise level, Maximum range, Border size
* '''Output''': Range image (spherical projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image.html pcl::RangeImage]
* [http://robotica.unileon.es/~victorm/PCL_range_image_spherical.tar.gz Download]
</div>

====Planar projection====

As mentioned, planar projection will give better results with clouds taken from a depth camera:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the planar range image object:

// Image size. Both Kinect and Xtion work at 640x480.
int imageSizeX = 640;
int imageSizeY = 480;
// Center of projection. here, we choose the middle of the image.
float centerX = 640.0f / 2.0f;
float centerY = 480.0f / 2.0f;
// Focal length. The value seen here has been taken from the original depth images.
// It is safe to use the same value vertically and horizontally.
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;

// Planar range image object.
pcl::RangeImagePlanar rangeImagePlanar;
rangeImagePlanar.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Planar range image");
viewer.showRangeImage(rangeImagePlanar);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_planar.png|thumb|center|400px|Range image of a point cloud, using planar projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Image size, Projection center, Focal length, Sensor pose, Coordinate frame, Noise level, Maximum range
* '''Output''': Range image (planar projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_planar.html pcl::RangeImagePlanar]
* [http://robotica.unileon.es/~victorm/PCL_range_image_planar.tar.gz Download]
</div>

If you prefer to do the conversion in real time while you inspect the cloud, PCL ships with an [https://github.com/PointCloudLibrary/pcl/tree/master/doc/tutorials/content/sources/openni_range_image_visualization example] that fetches an <span style="color:#FF1493">"openni_wrapper::DepthImage"</span> from an OpenNI device and creates the range image from it. You can adapt the code of the [[PCL/OpenNI_tutorial_1:_Installing_and_testing#Testing | example]] example from tutorial 1 to save it to disk with the function [http://docs.pointclouds.org/trunk/group__io.html#ga7291a029cdcde32ca3639d07dc6491b9 pcl::io::saveRangeImagePlanarFilePNG()].

===Extracting borders===

NARF keypoints are located near the edges of objects in the range image, so in order to find them, we first have to extract the borders. A border is defined as an abrupt change from foreground to background. In a range image, this can be easily seen because there is a "jump" in the depth value of two adjacent pixels.

[[Image:Range_image_border_detection.png|thumb|center|400px|Border detection on a range image (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

There are three types of borders. ''Object borders'' consist of the pixels (or points) located on the very edge of an object (the outermost points still belonging to the object). ''Shadow borders'' are points in the background on the edge of occlusions (empty areas in the background due to the objects in front covering them). Notice that, when the cloud is seen from the sensor's viewpoint, object and shadow points will seem adjacent. Finally, ''veil points'' are points interpolated between the previous two which appear in scans taken with LIDAR sensors, so we do not have to worry about them here.

<center><gallery widths=300px>
File:Range_image_border_type.png | Types of borders (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:Range_image_borders_example.png | Example of object and shadow borders on a cloud.
</gallery></center>

The algorithm basically compares a point's depth with the values of its neighbors, and if a big difference is found, we know it is due to a border. Points closer to the sensor will be marked as object borders, and the other ones as shadow borders.

PCL provides a class for extracting borders of a range image:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the borders.
pcl::PointCloud<pcl::BorderDescription>::Ptr borders(new pcl::PointCloud<pcl::BorderDescription>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Border extractor object.
pcl::RangeImageBorderExtractor borderExtractor(&rangeImage);

borderExtractor.compute(*borders);

// Visualize the borders.
pcl::visualization::RangeImageVisualizer* viewer = NULL;
viewer = pcl::visualization::RangeImageVisualizer::getRangeImageBordersWidget(rangeImage,
-std::numeric_limits<float>::infinity(),
std::numeric_limits<float>::infinity(),
false, *borders, "Borders");

while (!viewer->wasStopped())
{
viewer->spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_borders.png|thumb|center|400px|Borders found on the range image.]]

You can use the extractor's <span style="color:#FF1493">"getParameters()"</span> function to get a [http://docs.pointclouds.org/trunk/structpcl_1_1_range_image_border_extractor_1_1_parameters.html pcl::RangeImageBorderExtractor::Parameters] struct with the settings that will be used.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image
* '''Output''': Borders
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/range_image_border_extraction.php How to extract borders from range images]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_border_extractor.html pcl::RangeImageBorderExtractor]
* [http://robotica.unileon.es/~victorm/PCL_range_image_borders.tar.gz Download]
</div>

===Finding keypoints===

Citing the [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original publication]:''

''"We have the following requirements for our interest point extraction procedure:

# ''It must take information about borders and the surface structure into account.''
# ''It must select positions that can be reliably detected even if the object is observed from another perspective.''
# ''The points must be on positions that provide stable areas for normal estimation or the descriptor calculation in general."''

The procedure is the following: for every point in the range image, a score is computed that conveys how much the surface changes in its neighborhood (this is tuned with the ''support size'' σ, which is the diameter of the sphere used to find neighboring points). Also, the dominant direction of this change is computed. Then, this direction is compared with those of the neighbors, trying to find how stable the point is (if the directions are very different, that means the point is not stable, and that the surface around changes a lot). Points that are near the object's corners (but not exactly on the very edge) will be good keypoints, yet stable enough.

<center><gallery widths=300px>
File:NARF_keypoints.png | NARF keypoints (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:NARF_keypoints_support_sizes.png | Interest regions with a support size of 20cm (up) and 1m (down) (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
</gallery></center>

In PCL, NARF keypoints can be found this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

pcl::RangeImageBorderExtractor borderExtractor;
// Keypoint detection object.
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
// The support size influences how big the surface of interest will be,
// when finding keypoints from the border information.
detector.getParameters().support_size = 0.2f;

detector.compute(*keypoints);

// Visualize the keypoints.
pcl::visualization::RangeImageVisualizer viewer("NARF keypoints");
viewer.showRangeImage(rangeImage);
for (size_t i = 0; i < keypoints->points.size(); ++i)
{
viewer.markPoint(keypoints->points[i] % rangeImage.width,
keypoints->points[i] / rangeImage.width,
// Set the color of the pixel to red (the background
// circle is already that color). All other parameters
// are left untouched, check the API for more options.
pcl::visualization::Vector3ub(1.0f, 0.0f, 0.0f));
}

while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_NARF_keypoints.png|thumb|center|400px|NARF keypoints found on the range image.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Border extractor, Support Size
* '''Output''': NARF keypoints
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_keypoint_extraction.php How to extract NARF keypoint from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_keypoint.html pcl::NarfKeypoint]
* [http://robotica.unileon.es/~victorm/PCL_NARF_keypoints.tar.gz Download]
</div>

===Computing the descriptor===

We have created the range image from a point cloud, and we have extracted the borders in order to find good keypoints. Now it is time to compute the NARF descriptor for each keypoint.

The NARF descriptor encodes information about surface changes around a point. First, a local range patch is created around the point. It is like a small range image centered at that point, aligned with the normal (it would seem as if we were looking at the point along the normal). Then, a star pattern with ''n'' beams is overlaid onto the patch, also centered at the point. For every beam, a value is computed, that reflects how much the surface under it changes. The stronger the change is, and the closer to the center it is, the higher the final value will be. The ''n'' resulting values compose the final descriptor.

[[Image:NARF_descriptor.png|thumb|center|600px|Computing the NARF descriptor for a keypoint (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

The descriptor right now is not invariant to rotations around the normal. To achieve this, the whole possible 360 degrees are binned into an histogram. The value of each bin is computed from the descriptor values according to the angle. Then, the bin with the highest value is considered the dominant orientation, and the descriptor is shifted according to it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/features/narf_descriptor.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);
// Object for storing the NARF descriptors.
pcl::PointCloud<pcl::Narf36>::Ptr descriptors(new pcl::PointCloud<pcl::Narf36>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Extract the keypoints.
pcl::RangeImageBorderExtractor borderExtractor;
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
detector.getParameters().support_size = 0.2f;
detector.compute(*keypoints);

// The NARF estimator needs the indices in a vector, not a cloud.
std::vector<int> keypoints2;
keypoints2.resize(keypoints->points.size());
for (unsigned int i = 0; i < keypoints->size(); ++i)
keypoints2[i] = keypoints->points[i];
// NARF estimation object.
pcl::NarfDescriptor narf(&rangeImage, &keypoints2);
// Support size: choose the same value you used for keypoint extraction.
narf.getParameters().support_size = 0.2f;
// If true, the rotation invariant version of NARF will be used. The histogram
// will be shifted according to the dominant orientation to provide robustness to
// rotations around the normal.
narf.getParameters().rotation_invariant = true;

narf.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Key points, Support Size
* '''Output''': NARF descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_feature_extraction.php How to extract NARF Features from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_descriptor.html pcl::NarfDescriptor]
* [http://robotica.unileon.es/~victorm/PCL_NARF_descriptor.tar.gz Download]
</div>

==RoPS==

The Rotational Projection Statistics (RoPS) feature is a bit different from the other descriptors because it works with a triangle mesh, so a previous triangulation step is needed for generating this mesh from the cloud. Apart from that, most concepts are similar.

In order to compute RoPS for a keypoint, the local surface is cropped according to a support radius, so only points and triangles lying inside are taken into account. Then, a local reference frame (LRF) is computed, giving the descriptor its rotational invariance. A coordinate system is created with the point as the origin, and the axes aligned with the LRF. Then, for every axis, several steps are performed.

First, the local surface is rotated around the current axis. The angle is determined by one of the parameters, which sets the number of rotations. Then, all points in the local surface are projected onto the XY, XZ and YZ planes. For each one, statistical information about the distribution of the projected points is computed, and concatenated to form the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/surface/gp3.h>
#include <pcl/features/rops_estimation.h>

// A handy typedef.
typedef pcl::Histogram<135> ROPS135;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing both the points and the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr cloudNormals(new pcl::PointCloud<pcl::PointNormal>);
// Object for storing the ROPS descriptor for each point.
pcl::PointCloud<ROPS135>::Ptr descriptors(new pcl::PointCloud<ROPS135>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Perform triangulation.
pcl::concatenateFields(*cloud, *normals, *cloudNormals);
pcl::search::KdTree<pcl::PointNormal>::Ptr kdtree2(new pcl::search::KdTree<pcl::PointNormal>);
kdtree2->setInputCloud(cloudNormals);
pcl::GreedyProjectionTriangulation<pcl::PointNormal> triangulation;
pcl::PolygonMesh triangles;
triangulation.setSearchRadius(0.025);
triangulation.setMu(2.5);
triangulation.setMaximumNearestNeighbors(100);
triangulation.setMaximumSurfaceAngle(M_PI / 4); // 45 degrees.
triangulation.setNormalConsistency(false);
triangulation.setMinimumAngle(M_PI / 18); // 10 degrees.
triangulation.setMaximumAngle(2 * M_PI / 3); // 120 degrees.
triangulation.setInputCloud(cloudNormals);
triangulation.setSearchMethod(kdtree2);
triangulation.reconstruct(triangles);

// Note: you should only compute descriptors for chosen keypoints. It has
// been omitted here for simplicity.

// RoPs estimation object.
pcl::ROPSEstimation<pcl::PointXYZ, ROPS135> rops;
rops.setInputCloud(cloud);
rops.setSearchMethod(kdtree);
rops.setRadiusSearch(0.03);
rops.setTriangles(triangles.polygons);
// Number of partition bins that is used for distribution matrix calculation.
rops.setNumberOfPartitionBins(5);
// The greater the number of rotations is, the bigger the resulting descriptor.
// Make sure to change the histogram size accordingly.
rops.setNumberOfRotations(3);
// Support radius that is used to crop the local surface of the point.
rops.setSupportRadius(0.025);

rops.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Triangles, Search method, Support radius, Number of rotations, Number of partition bins
* '''Output''': RoPS descriptors
* '''Publication''':
** [http://arxiv.org/pdf/1304.3192v1.pdf Rotational Projection Statistics for 3D Local Surface Description and Object Recognition] (Yulan Guo et al., 2013)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_o_p_s_estimation.html pcl::ROPSEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RoPS.tar.gz Download]
</div>

=Global descriptors=

Global descriptors encode object geometry. They are not computed for individual points, but for a whole cluster that represents an object. Because of this, a preprocessing step ([[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]]) is required, in order to retrieve possible candidates.

Global descriptors are used for object recognition and classification, geometric analysis (object type, shape...), and pose estimation.

You should also know that many local descriptors can also be used as global ones. This can be done with descriptors that use a radius to search for neighbors (as PFH does). The trick is to compute it for one single point in the object cluster, and set the radius to the maximum possible distance between any two points (so all points in the cluster are considered as neighbors).

==VFH==

The Viewpoint Feature Histogram is based on the FPFH. Because the latter is invariant to the object's pose, the authors decided to expand it by including information about the viewpoint. Also, the FPFH is estimated once for the whole cluster, not for every point.

The VFH is made up by two parts: a viewpoint direction component, and an extended FPFH component. To compute the first one, the object's [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]] is found, which is the point that results from averaging the X, Y and Z coordinates of all points. Then, the vector between the viewpoint (the position of the sensor) and this centroid is computed, and normalized. Finally, for all points in the cluster, the angle between this vector and their normal is calculated, and the result is binned into an histogram. The vector is translated to each point when computing the angle because it makes the descriptor scale invariant.

The second component is computed like the FPFH (that results in 3 histograms for the 3 angular features, α, φ and θ), with some differences: it is only computed for the centroid, using the computed viewpoint direction vector as its normal (as the point, obviously, does not have a normal), and setting all the cluster's points as neighbors.

<center><gallery widths=300px>
File:VFH_viewpoint_component.png | Viewpoint component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
File:VFH_extended_FPFH_component.png | Extended FPFH component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
</gallery></center>

The resulting 4 histograms (1 for the viewpoint component, 3 for the extended FPFH component) are concatenated to build the final VFH descriptor. By default, the bins are normalized using the total number of points in the cluster. This makes the VFH descriptor invariant to scale.

[[Image:VFH_histogram.png|thumb|center|400px| VFH histogram (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).]]

The PCL implementation computes an additional fifth histogram with the distances of the cluster points to the centroid (the Shape Distribution Component, SDC), increading the size of the output descriptor from 263 to 308. The SDC is taken from the CVFH descriptor that we will see in the next section, and makes the result more robust.

The VFH of an already clustered object can be computed this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the VFH descriptor.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// VFH estimation object.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
// Optionally, we can normalize the bins of the resulting histogram,
// using the total number of points.
vfh.setNormalizeBins(true);
// Also, we can normalize the SDC with the maximum size found between
// the centroid and any of the cluster's points.
vfh.setNormalizeDistance(false);

vfh.compute(*descriptor);
}</syntaxhighlight>

Because only one VFH descriptor is computed for the whole cluster, the size of the cloud object that stores the result will be 1.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, [Normalize bins], [Normalize SDC]
* '''Output''': VFH descriptor
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_estimation.php Estimating VFH signatures for a set of points]
* '''Publication''':
** [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf Fast 3D Recognition and Pose Using the Viewpoint Feature Histogram] (Radu Bogdan Rusu et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_v_f_h_estimation.html pcl::VFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_VFH.tar.gz Download]
</div>

===CVFH===

The original VFH descriptor is not robust to occlusion or other sensor artifacts, or measurement errors. If the object cluster is missing many points, the resulting computed centroid will differ from the original, altering the final descriptor, and preventing a positive match from being found. Because of that, the Clustered Viewpoint Feature Histogram (CVFH) was introduced.

The idea is very simple: instead of computing a single VFH histogram for the whole cluster, the object is first divided in stable, smooth regions using [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Region_growing | region-growing segmentation]], that enforces several constraints in the distances and differences of normals of the points belonging to every region. Then, a VFH is computed for every region. Thanks to this, an object can be found in a scene, as long as at least one of its regions is fully visible.

<center><gallery widths=300px>
File:CVFH_occlusion.png | Typical occlusion issues in a point cloud (image from original paper).
File:CVFH_regions.png | Object regions computed for the CVFH (image from original paper).
</gallery></center>

Additionally, a Shape Distribution Component (SDC) is also computed and included. It encodes information about the distribution of the points arond the region's centroid, measuring the distances. The SDC allows to differentiate objects with similar characteristics (size and normal distribution), like two planar surfaces from each other.

The authors proposed to discard the histogram normalization step that is performed in VFH. This has the effect of making the descriptor dependant of scale, so an object of a certain size would not match a bigger or smaller copy of itself. It also makes CVFH more robust to occlusion.

CVFH is invariant to the camera roll angle, like most global descriptors. This is so because rotations about that camera axis do not change the observable geometry that descriptors are computed from, limiting the pose estimation to 5 DoF. The use of a Camera Roll Histogram (CRH) has been proposed to overcome this.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CVFH estimation object.
pcl::CVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> cvfh;
cvfh.setInputCloud(object);
cvfh.setInputNormals(normals);
cvfh.setSearchMethod(kdtree);
// Set the maximum allowable deviation of the normals,
// for the region segmentation step.
cvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
// Set the curvature threshold (maximum disparity between curvatures),
// for the region segmentation step.
cvfh.setCurvatureThreshold(1.0);
// Set to true to normalize the bins of the resulting histogram,
// using the total number of points. Note: enabling it will make CVFH
// invariant to scale just like VFH, but the authors encourage the opposite.
cvfh.setNormalizeBins(false);

cvfh.compute(*descriptors);
}</syntaxhighlight>

You can further customize the segmentation step with <span style="color:#FF1493">"setClusterTolerance()"</span> (to set the maximum Euclidean distance between points in the same cluster) and <span style="color:#FF1493">"setMinPoints()"</span>. The size of the output will be equal to the number of regions the object was divided in. Also, check the functions <span style="color:#FF1493">"getCentroidClusters()"</span> and <span style="color:#FF1493">"getCentroidNormalClusters()"</span>, you can use them to get information about the centroids used to compute the different CVFH descriptors.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points]
* '''Output''': CVFH descriptors
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_v_f_h_estimation.html pcl::CVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CVFH.tar.gz Download]
</div>

====OUR-CVFH====

The Oriented, Unique and Repeatable CVFH expands the previous descriptor, adding the computation of an unique reference frame to make it more robust.

OUR-CVFH relies on the use of Semi-Global Unique Reference Frames (SGURFs), which are repeatable coordinate systems computed for each region. Not only they remove the invariance to camera roll and allow to extract the 6DoF pose directly without additional steps, but they also improve the spatial descriptiveness.

The first part of the computation is akin to CVFH, but after segmentation, the points in each region are filtered once more according to the difference between their normals and the region's average normal. This results in better shaped regions, improving the estimation of the Reference Frames (RFs).

After this, the SGURF is computed for each region. Disambiguation is performed to decide the sign of the axes, according to the points' distribution. If this is not enough and the sign remains ambiguous, multiple RFs will need to be created to account for it. Finally, the OUR-CVFH descriptor is computed. The original Shape Distribution Component (SDC) is discarded, and the surface is now described according to the RFs.

[[Image:OUR-CVFH.png|thumb|center|600px| SGURF frame and resulting histogram of a region (image from [http://vision.deis.unibo.it/fede/papers/dagm12.pdf original paper]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/our_cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the OUR-CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// OUR-CVFH estimation object.
pcl::OURCVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> ourcvfh;
ourcvfh.setInputCloud(object);
ourcvfh.setInputNormals(normals);
ourcvfh.setSearchMethod(kdtree);
ourcvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
ourcvfh.setCurvatureThreshold(1.0);
ourcvfh.setNormalizeBins(false);
// Set the minimum axis ratio between the SGURF axes. At the disambiguation phase,
// this will decide if additional Reference Frames need to be created, if ambiguous.
ourcvfh.setAxisRatio(0.8);

ourcvfh.compute(*descriptors);
}</syntaxhighlight>

You can use the <span style="color:#FF1493">"getTransforms()"</span> function to get the transformations aligning the cloud to the corresponding SGURF.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points], [Axis ratio]
* '''Output''': OUR-CVFH descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/dagm12.pdf OUR-CVFH – Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram for Object Recognition and 6DOF Pose Estimation] (Aitor Aldoma et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_o_u_r_c_v_f_h_estimation.html pcl::OURCVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_OUR-CVFH.tar.gz Download]
</div>

==ESF==

The Ensemble of Shape Functions (ESF) is a combination of 3 different shape functions that describe certain properties of the cloud's points: distances, angles and area. This descriptor is very unique because it does not require normal information. Actually, it does not need any preprocessing, as it is robust to noise and incomplete surfaces.

The algorithm uses a voxel grid as an approximation of the real surface. It iterates through all the points in the cloud: for every iteration, 3 random points are chosen. For these points, the shape functions are computed:

* '''D2''': this function computes the distances between point pairs (3 overall). Then, for every pair, it checks if the line that connects both points lies entirely inside the surface, entirely outside (crossing free space), or both. Depending on this, the distance value will be binned to one of three possible histograms: IN, OUT or MIXED.
* '''D2 ratio''': an additional histogram for the ratio between parts of the line inside the surface, and parts outside. This value will be 0 if the line is completely outside, 1 if completely inside, and some value in between if mixed.
* '''D3''': this computes the square root of the area of the triangle formed by the 3 points. Like D2, the result is also classified as IN, OUT or MIXED, each with its own histogram.
* '''A3''': this function computes the angle formed by the points. Then, the value is binned depending on how the line opposite to the angle is (once again, as IN, OUT or MIXED).

[[Image:ESF.png|thumb|center|600px| ESF descriptor (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

After the loop is over, we are left with 10 subhistograms (IN, OUT and MIXED for D2, D3 and A3, and an additional one for the ratio). Each one has 64 bins, so the size of the final ESF descriptor is 640.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the ESF descriptor.
pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// ESF estimation object.
pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
esf.setInputCloud(object);

esf.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster)
* '''Output''': ESF descriptor
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6181760 Ensemble of shape functions for 3D object classification] (requires IEEE Xplore subscription) (Walter Wohlkinger and Markus Vincze, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_e_s_f_estimation.html pcl::ESFEstimation]
* [http://robotica.unileon.es/~victorm/PCL_ESF.tar.gz Download]
</div>

==GFPFH==

As you may have guessed, GFPFH stands for Global Fast Point Feature Histogram, the global version of the FPFH descriptor. GFPFH was designed for the task of helping a robot navigate its environment, having some context of the objects around it.

The first step before being able to compute the descriptor is surface categorization. A set of logical primitives (the classes, or categories) is created, which depends on the type of objects we expect the robot to find on the scene. For example, if we know there will be a coffee mug, we create three: one for the handle, and the other two for the outer and inner faces. Then, FPFH descriptors are computed, and everything is fed to a [https://en.wikipedia.org/wiki/Conditional_random_field Conditional Random Field] (CRF) algorithm. The CRF will label each surface with one of the previous categories, so we end up with a cloud where each point has been classified depending of the type of object (or object's region) it belongs to.

[[Image:FPFH_CRF.png|thumb|center|300px| Classification of objects made with FPFH and CRF (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

Now, the GFPFH descriptor can be computed with the result of the classification step. It will encode what the object is made of, so the robot can easily recognize it. First, an [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Octree | octree]] is created, dividing the object in voxel leaves. For every leaf, a set of probabilities is created, one for each class. Each one stores the probability of that leaf belonging to the class, and it is computed according to the number of points in that leaf that have been labelled as that class, and the total number of points. Then, for every pair of leaves in the octree, a line is casted, connecting them. Every leaf in its path is checked for occupancy, storing the result in an histogram. If the leaf is empty (free space), a value of 0 is saved. Otherwise, the leaf probabilities are used.

[[Image:GFPFH.png|thumb|center|500px| Computing the GFPFH with a voxel grid (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

The following code will compute the GFPFH for a cloud with label information. The categorization step is up to you, as it depends largely on the type of the scene, and the use you are going to give it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/gfpfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZL>::Ptr object(new pcl::PointCloud<pcl::PointXYZL>);
// Object for storing the GFPFH descriptor.
pcl::PointCloud<pcl::GFPFHSignature16>::Ptr descriptor(new pcl::PointCloud<pcl::GFPFHSignature16>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZL>(argv[1], *object) != 0)
{
return -1;
}

// Note: you should now perform classification on the cloud's points. See the
// original paper for more details. For this example, we will now consider 4
// different classes, and randomly label each point as one of them.
for (size_t i = 0; i < object->points.size(); ++i)
{
object->points[i].label = 1 + i % 4;
}

// ESF estimation object.
pcl::GFPFHEstimation<pcl::PointXYZL, pcl::PointXYZL, pcl::GFPFHSignature16> gfpfh;
gfpfh.setInputCloud(object);
// Set the object that contains the labels for each point. Thanks to the
// PointXYZL type, we can use the same object we store the cloud in.
gfpfh.setInputLabels(object);
// Set the size of the octree leaves to 1cm (cubic).
gfpfh.setOctreeLeafSize(0.01);
// Set the number of classes the cloud has been labelled with (default is 16).
gfpfh.setNumberOfClasses(4);

gfpfh.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Labels, Number of classes, Leaf size
* '''Output''': GFPFH descriptor
* '''Publication''':
** [https://www.willowgarage.com/sites/default/files/iccv09.pdf Detecting and Segmenting Objects for Mobile Manipulation] (Radu Bogdan Rusu et al., 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_f_p_f_h_estimation.html pcl::GFPFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GFPFH.tar.gz Download]
</div>

==GRSD==

The global version of the Radius-based Surface Descriptor works in a similar fashion to GFPFH. A voxelization and a surface categorization step are performed beforehand, labelling all surface patches according to the geometric category (plane, cylinder, edge, rim, sphere), using RSD. Then, the whole cluster is classified into one of these categories, and the GRSD descriptor is computed from this.

[[Image:GRSD.png|thumb|center|400px| Classification of objects for GRSD and resulting histogram (image from [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf original paper]).]]

To compute it:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/grsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the GRSD descriptors for each point.
pcl::PointCloud<pcl::GRSDSignature21>::Ptr descriptors(new pcl::PointCloud<pcl::GRSDSignature21>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// GRSD estimation object.
pcl::GRSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::GRSDSignature21> grsd;
grsd.setInputCloud(cloud);
grsd.setInputNormals(normals);
grsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
grsd.setRadiusSearch(0.05);

grsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Radius
* '''Output''': GRSD descriptor
* '''Publications''':
** [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
** [http://www.mi.t.u-tokyo.ac.jp/top/downloadpublication/36 Voxelized Shape and Color Histograms for RGB-D] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_r_s_d_estimation.html pcl::GRSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GRSD.tar.gz Download]
</div>

=Saving and loading=

You can save a descriptor to a file just [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Writing_to_file|like with any other cloud type]]. One caveat, though. If you are using a descriptor that has its own custom type, like <span style="color:#FF1493">"PFHSignature125"</span>, everything will be OK. But with descriptors that do not (where you have to use <span style="color:#FF1493">"pcl::Histogram<>"</span>), you will get this error: <span style="color:#FF1493">''"POINT_TYPE_NOT_PROPERLY_REGISTERED"''</span>. In order to save or load from a file, PCL's IO functions need to know about the number, type and size of the fields. To solve this, you will have to properly register a new point type for your descriptor. For example, this will work for the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#RoPS|RoPS]] descriptor example we saw earlier:

<syntaxhighlight lang=CPP>POINT_CLOUD_REGISTER_POINT_STRUCT(ROPS135,
(float[135], histogram, histogram)
)</syntaxhighlight>

Add the previous to your code (change it accordingly), and you will be able to save and load descriptors as usual.

=Visualization=

Sometimes it is desired to check a visual representation of a descriptor, perhaps to analyze the distribution of data over different bins. Because they are saved as histograms, this is something trivial to do. PCL offers a couple of classes to do this.

==PCLHistogramVisualizer==

<span style="color:#FF1493">"PCLHistogramVisualizer"</span> is the simplest way to plot an histogram. The class has little functionality, but it does its job. Only one call is necessary to give the histogram and its size:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/histogram_visualizer.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLHistogramVisualizer viewer;
// We need to set the size of the descriptor beforehand.
viewer.addFeatureHistogram(*descriptor, 308);

viewer.spin();
}</syntaxhighlight>

[[Image:Histogram_visualizer_VFH.png|thumb|center|500px| VFH histogram seen with the Histogram Visualizer.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_histogram_visualizer.html pcl::visualization::PCLHistogramVisualizer]
* [http://robotica.unileon.es/~victorm/PCL_histogram_visualizer.tar.gz Download]
</div>

==PCLPlotter==

This class has all the methods from <span style="color:#FF1493">"PCLHistogramVisualizer"</span> (which will be deprecated soon) plus a lot more features. The code is almost the same:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/pcl_plotter.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLPlotter plotter;
// We need to set the size of the descriptor beforehand.
plotter.addFeatureHistogram(*descriptor, 308);

plotter.plot();
}</syntaxhighlight>

[[Image:PCL_plotter_VFH.png|thumb|center|500px| VFH histogram seen with the PCLPlotter class.]]

If you have raw data (such as a vector of floats) you can use the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html#aaf23b2b1c2f91c517cfde387ee1b654e addHistogramData()] function to plot it as an histogram.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pcl_plotter.php PCLPlotter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html pcl::visualization::PCLPlotter]
* [http://robotica.unileon.es/~victorm/PCL_plotter.tar.gz Download]
</div>

==PCL Viewer==

This program, included with PCL, will also let you open and visualize a saved descriptor. Internally, it uses [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#PCLPlotter|PCLPlotter]]. You can invoke the viewer from the command line like this, so it comes handy:

<syntaxhighlight lang=Bash>pcl_viewer <descriptor_file></syntaxhighlight>

[[Image:PCL_viewer_VFH.png|thumb|center|500px| VFH histogram seen with ''pcl_viewer''.]]

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 1: Installing and testing

2015-11-05T16:53:34Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Kinect.jpg|thumb|right|200px|Microsoft Kinect device.]]

This series of tutorials will explain the usage of a depth camera like [https://en.wikipedia.org/wiki/Kinect Kinect] for "serious" researching purposes. As you may know, Kinect is in fact an affordable depth sensor, developed with technology from [https://en.wikipedia.org/wiki/PrimeSense PrimeSense], based on infrarred structured light method. It also has a common camera (which makes it a RGB-D device), a microphone and a motorized pivot. Its use is not limited to playing with a Xbox360 console, you can plug it to a computer and use it like any other sensor. Many open-source drivers and frameworks are available.

Since its release on November 2010, it has gained a lot of popularity, specially among the scientific community. Many researches have procured themselves one because, despite the low cost (about 100 $), it has proven to be a powerful solution for depth sensing projects. Current investigations focus on real-time surface mapping, object recognition and tracking, and localization. Impressive results (like the [http://research.microsoft.com/en-us/projects/surfacerecon/ KinectFusion] project from Microsoft) are already possible.

The new [https://en.wikipedia.org/wiki/Xbox_One Xbox One] ships with an upgraded version, [https://en.wikipedia.org/wiki/Kinect_for_Xbox_One Kinect v2], with enhanced resolution, that is able to detect your facial expression, measure your heart rate, and track every one of your fingers. A PC development-ready version ([https://en.wikipedia.org/wiki/Kinect_for_Xbox_One#Kinect_for_Windows Kinect for Windows v2]) was released in July 2014, but it could only be used with the official Windows SDK (open source support [https://github.com/OpenKinect/libfreenect2 exists] but is still young). In October 2014 an adapter that allows to plug the standard Kinect v2 to a PC was released, so the development version of the sensor was discontinued in April 2015. Now you can just buy the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-for-Xbox-One/productID.307499400 standard one] for the console plus the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600 adapter].

I will explain the installation and usage of one of the "legacy" Kinect 1.0 devices with a common PC, and the possibilities it offers. I will do it in an easy to understand way, intended for students that have just acquired it and want to start from scratch.

Keep in mind that the software that we are going to use (Point Cloud Library and OpenNI drivers) will also let you use any other device like the [https://www.asus.com/3D-Sensor/Xtion_PRO/ Xtion PRO] or [https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/ Xtion PRO LIVE] from ASUS (the PRO only has a depth sensor, the PRO LIVE is a RGB-D camera) without changing a line of code.

<span style="color:#606060">'''''NOTE: The tutorials are written for Linux platforms. Also, 64-bit versions seem to work better than 32-bit.'''''</span>

=Requirements=

You will need the following:

* A common Kinect device, out of the box. You can buy it in your local electronics shop, or online. It also includes a free copy of ''[https://en.wikipedia.org/wiki/Kinect_Adventures! Kinect Adventures!]'', which is useless if you do not own the console. Microsoft has released a ''Kinect for Windows'' device, which is a normal looking Kinect no longer compatible with Xbox360, that will only work with their official SDK, intended for developers only. Also, like I stated earlier, you can use an ASUS Xtion indistinctly.
* A computer running Linux (Debian or Ubuntu preferably).
* A medium-sized room. Kinect has some limitations for depth measurement: 40cm minimum, 8m maximum (make it 6).

<span style="color:#606060">'''''NOTE: ''Kinect for Windows'' may have problems working with open source drivers on Linux .'''''</span>

=Connecting everything=

Kinect does not work with a common USB port. Its power consumption is a bit higher because of the motor, so Microsoft came up with a connector that combines USB and power supply. Old Xbox 360 models needed a special adapter, new S model ones already have this new port. Luckily, Kinect comes with the official adapter out of the box (otherwise you will have to [http://www.amazon.com/Power-Supply-Cable-Kinect-Xbox-360/dp/B004S7GA46/ref=sr_1_1?ie=UTF8&qid=1378288327&sr=8-1 buy one]).

Just plug the adapter to any power socket, and the USB to your computer. Let's check typing this in a terminal:

<syntaxhighlight lang=Bash>lsusb</syntaxhighlight>

Output should list the following devices:

<syntaxhighlight lang=Bash>Bus 001 Device 005: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 006: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 007: ID 045e:02ae Microsoft Corp. Xbox NUI Camera</syntaxhighlight>

If you are using a Xtion, you should see:

<syntaxhighlight lang=Bash>Bus 001 Device 004: ID 1d27:0601 ASUS</syntaxhighlight>

"0601" identifies the new Xtion model, "0600" the older one. Both should work the same. But try to avoid [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO|USB 3.0 ports]]!

=Installing the software=

There is more than one way to get your Kinect working on your PC, and start developing applications for it:

* [https://www.microsoft.com/en-us/download/details.aspx?id=40278 Kinect for Windows SDK] and [https://www.microsoft.com/en-us/download/details.aspx?id=40276 Developer Toolkit]: released on June 16, 2011 as a non-commercial SDK intended for application development. Version 1.8, the last there will ever be now that Kinect v2 is out, was released on September 2013. Because it comes from Microsoft, it is obviously the easiest way to get everything working. Sadly, there is no Linux version.
* [https://github.com/OpenKinect/libfreenect libfreenect] library: from the [http://openkinect.org/wiki/Main_Page OpenKinect] project, it is intended to be a free and open source alternative to the official drivers. libfreenect is used by projects like [https://github.com/ofTheo/ofxKinect ofxKinect], an addon for the [http://www.openframeworks.cc/ openFrameworks] toolkit (and as of version 0.8, included in the core package) that runs on Linux and OS X. ofxKinect packs a nice example application to show the RGB and point cloud taken from Kinect.
* [https://github.com/PrimeSense/Sensor PrimeSense drivers]: they were released as open source after the the [https://en.wikipedia.org/wiki/OpenNI OpenNI] project was created, along with the motion tracking middleware, ''NITE'', and the SDK. NI stands for Natural Interaction, and the project tried to enforce a common standard for human input using Kinect-like sensors. These official drivers are used by [http://wiki.ros.org/openni_kinect ROS] (the Robot Operating System, a massive collection of libraries and tools for robotic researchers) and [http://pointclouds.org/ PCL] (the Point Cloud Library, with everything needed for 3D point cloud processing). Sadly, version 2.0 of the OpenNI SDK dropped support for Kinect on Linux due to licensing issues, so for now PCL relies on legacy 1.x versions. Also, Apple bought PrimeSense on November 2013, and on April 2014 OpenNI's webpage was closed. The source is now being maintained by [http://structure.io/openni a third party].
* [https://github.com/avin2/SensorKinect SensorKinect]: a modified version of the official PrimeSense drivers, used for example by [https://github.com/gameoverhack/ofxOpenNI ofxOpenNI] (another openFrameworks addon). Last updated on 2012.

For this tutorial, we are going to use PCL with the OpenNI drivers, so owners of a Xtion can also get it to work.

==Precompiled PCL for Ubuntu==

If you have a valid [http://wiki.ros.org/ROS/Installation installation of ROS] (through their repository), you do not have to install anything. Both OpenNI and PrimeSense drivers, as well as PCL, should be already installed. You can check it with:

<syntaxhighlight lang=Bash>sudo apt-get install libpcl-1.7-all libpcl-1.7-all-dev libopenni-dev libopenni-sensor-primesense-dev</syntaxhighlight>

The previous command should state that all packages are already installed (change the PCL version number as needed), install them if not. If you get an error about overwriting some file, check [[PCL/OpenNI_troubleshooting#Trying_to_overwrite_X.2C_which_is_also_in_package_Y|this]].

In case you do not want ROS, there is a [https://launchpad.net/~v-launchpad-jochen-sprickerhof-de/+archive/ubuntu/pcl PPA] (Personal Package Archive, a private repository) which has everything we need. Add it to your sources, and install everything:

<syntaxhighlight lang=Bash>sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install build-essential cmake libpcl1.7 libpcl-dev pcl-tools</syntaxhighlight>

Trying to mix ROS and PCL repositories and packages can cause some errors, so choose one of them and stick with it. Check the [[PCL/OpenNI troubleshooting]] page because your Kinect may not work by default in 32 bits.

==Compiling PCL from source==

For Linuxes without a precompiled version of PCL, you will need to compile it yourself. This has an advantage, actually: you can customize the build options and choose what you want. Also, the resulting binaries and libraries should be a bit faster. And you always get the latest version! The instructions are [http://pointclouds.org/documentation/tutorials/compiling_pcl_posix.php here], but the steps are easy so I will show them to you.

===Installing the dependencies===

Some of PCL dependencies can be installed via the package manager. Others will require additional work.

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install build-essential cmake cmake-curses-gui libboost-all-dev libeigen3-dev libflann-dev libvtk5-dev libvtk5-qt4-dev libglew-dev libxmu-dev libsuitesparse-dev libqhull-dev libpcap-dev libxmu-dev libxi-dev libgtest-dev libqt4-opengl-dev</syntaxhighlight>

The trunk (1.8) version of PCL adds support for [https://en.wikipedia.org/wiki/VTK VTK] 6 and [https://en.wikipedia.org/wiki/Qt_%28software%29 Qt] 5, so if you want to use them, you must install the following packages (say yes if you are asked to remove VTK 5 and Qt 4 packages):

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

PCL will not build if you mix them, e.g., using VTK 5 with Qt 5 will give you an error halfway through the compilation.

====JDK====

OpenNI requires Sun's official JDK (Java Development Kit), which is no longer available on official apt repositories. You can use [http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html unofficial ones], or do it manually. For the last method, go to the [http://www.oracle.com/technetwork/java/javase/downloads/index.html Java SE downloads] page (SE means Standard Edition) and download the latest version (e.g., <span style="color:#1E90FF">''jdk-8u66-linux-x64.tar.gz''</span>). Extract it, then move the contents to <span style="color:#FF8C00">''/usr/lib/jvm/''</span> so it is available system-wide:

<syntaxhighlight lang=Bash>sudo mkdir -p /usr/lib/jvm/
sudo cp -r jdk1.8.0_66/ /usr/lib/jvm/</syntaxhighlight>

Then, make it the default choice to compile and run Java programs. Remember to change the version number as needed!

<syntaxhighlight lang=Bash>sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_66/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_66/bin/javac" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.8.0_66/bin/jar" 1</syntaxhighlight>

To be sure, use the following commands to manually select the correct option, in case there is more than one choice:

<syntaxhighlight lang=Bash>sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config jar</syntaxhighlight>

If you are still not sure, run them and display the version, making sure it is the one you installed:

<syntaxhighlight lang=Bash>java -version
javac -version</syntaxhighlight>

Sun's JDK is now installed.

====OpenNI====

PCL uses OpenNI and the PrimeSense drivers to get data from devices like Kinect or Xtion. It is optional, but it would not make much sense not to install it, would it? If you are using Ubuntu, add the [[#Precompiled PCL for Ubuntu | PPA]] and install <span style="color:#228B22">''libopenni-dev''</span> and <span style="color:#228B22">''libopenni-sensor-primesense-dev''</span>, which should be done already. Otherwise, fetch the [https://github.com/OpenNI/OpenNI OpenNI] and [https://github.com/PrimeSense/Sensor PrimeSense Sensor] sources from GitHub (download them as .zip, the link is on the right). Extract them, and install the dependencies:

<syntaxhighlight lang=Bash>sudo apt-get install python libusb-1.0-0-dev freeglut3-dev doxygen graphviz</syntaxhighlight>

Go to the directory where you extracted OpenNI (<span style="color:#FF8C00">''OpenNI-master/''</span> for me), and open a terminal in the <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span> subdirectory. Issue:

<syntaxhighlight lang=Bash>./RedistMaker</syntaxhighlight>

When it finishes, and if there are no errors (check the [[PCL/OpenNI troubleshooting]] page if you get any), go to <span style="color:#FF8C00">''Platform/Linux/Redist/OpenNI-Bin-Dev-Linux-x64-v1.5.7.10/''</span> (or your equivalent), and install (you must be root):

<syntaxhighlight lang=Bash>sudo ./install.sh</syntaxhighlight>

Now, go to the directory where you extracted the PrimeSense drivers (<span style="color:#FF8C00">''Sensor-master/''</span> for me), and repeat the exact same procedure (go to <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span>, issue <span style="color:#228B22">''./RedistMaker''</span>, go to <span style="color:#FF8C00">''Platform/Linux/Redist/Sensor-Bin-Linux-x64-v5.1.6.6/''</span>, issue <span style="color:#228B22">''sudo ./install.sh''</span>). Congratulations, you have now installed OpenNI.

====CUDA====

Like OpenNI, [https://en.wikipedia.org/wiki/CUDA nVidia CUDA] is an optional dependency, that will allow PCL to use your GPU (Graphics Processing Unit, that is, your graphics card) for certain computations. This is mandatory for tools like KinFu (do not bother unless you have at least a series 400 card with 1.5 GB of VRAM).

Some distributions provide packages for CUDA in the repositories. For example, in Ubuntu:

<syntaxhighlight lang=Bash>sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit</syntaxhighlight>

If you want to install it manually (which is incompatible with the previous method), go to the [https://developer.nvidia.com/cuda-downloads CUDA downloads] page, which is self-explanatory, and get the small .deb file or the huge .run toolkit/SDK installer file for your system (you should have installed the nVidia drivers already, but the installer will also do it for you if needed).

If you chose the .deb file, install it using the method of your choice, like Gdebi package manager, or through console:

<syntaxhighlight lang=Bash>sudo dpkg -i <package.deb></syntaxhighlight>

The .deb does not contain all CUDA stuff, it just adds their repository to your software lists. Now you must install everything:

<syntaxhighlight lang=Bash>sudo apt-get update
sudo apt-get install cuda</syntaxhighlight>

If, on the other hand, you downloaded the .run, give it execution permissions:

<syntaxhighlight lang=Bash>chmod +x cuda_7.0.28_linux.run</syntaxhighlight>

And install it. You can use the default options, although if you have a working nVidia graphics driver for your card, you may want to say "no" when the installer offers to install it for you:

<syntaxhighlight lang=Bash>sudo ./cuda_7.0.28_linux.run</syntaxhighlight>

The environment variables need to be changed so your system can find CUDA's libraries and binaries. Open <span style="color:#1E90FF">''/etc/ld.so.conf''</span>:

<syntaxhighlight lang=Bash>sudo nano /etc/ld.so.conf</syntaxhighlight>

And append one of these two lines:

<syntaxhighlight lang=Bash>/usr/local/cuda/lib64 # Add this on 64-bit only.
/usr/local/cuda/lib # Add this on 32-bit only.</syntaxhighlight>

Save with <span style="color:#228B22">'''Ctrl+O'''</span> and Enter, exit with <span style="color:#228B22">'''Ctrl+X'''</span>. Reload the cache of the dynamic linker with:

<syntaxhighlight lang=Bash>sudo ldconfig</syntaxhighlight>

Now, append CUDA's bin directory to your PATH. Do this by editing your local <span style="color:#1E90FF">''.bashrc''</span> file:

<syntaxhighlight lang=Bash>nano ~/.bashrc</syntaxhighlight>

And append this line:

<syntaxhighlight lang=Bash>export PATH=$PATH:/usr/local/cuda/bin</syntaxhighlight>

CUDA is now installed.

===Getting the source===

Every dependency is installed. Time to download PCL's source code. First, you must choose whether to install the stable or the experimental branch of PCL. The stable branch is the latest official release and it is guaranteed to work without problems. The experimental branch may give you a compiling error seldomly, but you can find some interesting features that stable users will have to wait some months for. Apart from that, both are built the same way.

To get the stable version, go to the [https://github.com/PointCloudLibrary/pcl/releases downloads] page, get <span style="color:#1E90FF">''pcl-pcl-1.7.2.tar.gz''</span> or whatever the latest release is, and extract it somewhere. For the experimental version, use Git:

<syntaxhighlight lang=Bash>sudo apt-get install git
git clone https://github.com/PointCloudLibrary/pcl PCL-trunk-Source</syntaxhighlight>

The compiled trunk version of PCL will take up more than 8 GB. So make sure you put the source folder in a partition with enough free space!

===Compiling===

Go the the PCL source directory (<span style="color:#FF8C00">''pcl-pcl-1.7.2/''</span> or <span style="color:#FF8C00">''PCL-trunk-Source/''</span>), and create a new subdirectory to keep the build files in:

<syntaxhighlight lang=Bash>mkdir build
cd build</syntaxhighlight>

Now it is time to configure the project using CMake. We will tell it to build in Release (fully optimized, no debug capabilities) mode now, and customize the rest of the options later:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..</syntaxhighlight>

CMake should be able to find every dependency, thus being able to build every subsystem except for the ones marked as <span style="color:#FF1493">''"Disabled by default"''</span>. If you are happy, you can build now, otherwise let's invoke CMake's curses interface to change a couple of things (mind the final dot):

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

[[Image:Ccmake_GUI_PCL.png|thumb|center|900px|Interface of ''ccmake''.]]

Here you can change the build options. The program usage can be found at the bottom of the screen. Try turning all functionality <span style="color:#FF1493">"ON"</span>. The most important thing, in case you want to use CUDA, is to enable it and give CMake the path to your SDK. Go to the <span style="color:#FF1493">"CUDA_SDK_ROOT_DIR"</span> option and enter the correct path (probably <span style="color:#FF8C00">''/usr/local/cuda/''</span> or similar).

When you are done, press <span style="color:#228B22">'''C'''</span> to configure and <span style="color:#228B22">'''G'''</span> to generate and exit the tool. Sometimes, the options you change can activate previously omitted parameters, or prompt some warning text. Just press <span style="color:#228B22">'''E'''</span> when you are finished reading the message, and keep pressing <span style="color:#228B22">'''C'''</span> until it lets you generate (new parameters will be marked with an asterisk, so you can check them and decide whether or not you want further customization).

If you are done configuring, it is time to build:

<syntaxhighlight lang=Bash>make</syntaxhighlight>

<span style="color:#606060">'''''NOTE: Additionally, you can append the parameter -jX to speed up the compilation, X being the number of cores or processors of your PC, plus one.'''''</span>

Remember that, at any time, you can manually force the project to be reconfigured and built from scratch by emptying the <span style="color:#FF8C00">''build/''</span> directory with:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

===Installing===

It will take some time to compile PCL (up to a few hours if your PC is not powerful enough). When it is finished, install it system-wide with:

<syntaxhighlight lang=Bash>sudo make install</syntaxhighlight>

And you should reboot and proceed to the next section, to see if your computer now recognizes (and uses) your Kinect device.

=Testing (OpenNI viewer)=

We are going to write a simple example program that will fetch data from the Kinect or Xtion and present it to the user, using the PCL library. It will also allow to save the current frame (as point cloud) to disk. If you feel lazy, you can download it [http://robotica.unileon.es/~victorm/PCL_openni_viewer.tar.gz here], and skip the next two sections. Otherwise, create a new directory anywhere in your hard disk and proceed.

==CMakeLists.txt==

Inside that directory, create a new text file named <span style="color:#1E90FF">''CMakeLists.txt''</span>. PCL-based programs use the CMake build system, too. Open it with any editor and paste the following content:

<syntaxhighlight lang=CMake>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

project(PCL_openni_viewer)

find_package(PCL 1.7 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

set(PCL_BUILD_TYPE Release)

file(GLOB PCL_openni_viewer_SRC
"src/*.h"
"src/*.cpp"
)
add_executable(openniViewer ${PCL_openni_viewer_SRC})

target_link_libraries (openniViewer ${PCL_LIBRARIES})</syntaxhighlight>

CMake syntax is quite self-explanatory. We ask for a CMake version 2.8 installation, minimum. We declare a new project named <span style="color:#FF1493">"openni_PCL_viewer"</span>. We tell CMake to check for the presence of PCL library development files, version 1.7. If our system can not meet the CMake and PCL version requirement, the process will fail.

Next, we feed the compiler and linker the directories where PCL includes and libraries can be found, and the defined symbols. We tell CMake to use the <span style="color:#FF1493">"Release"</span> build type, which will activate certain optimizations depending on the compiler we use. Other build types are available, like <span style="color:#FF1493">"Debug"</span>, <span style="color:#FF1493">"MinSizeRel"</span>, and <span style="color:#FF1493">"RelWithDebInfo"</span>.

Finally, we create a variable, <span style="color:#FF1493">"opennipclviewer_SRC"</span>, that will store a list of files to be compiled (though we will only have one). We create a new binary to be compiled from these source files, and we link it with the PCL library.

Check the [https://cmake.org/Wiki/CMake_Useful_Variables CMake help] for more interesting options.

==main.cpp==

We told CMake it could find the source files in a <span style="color:#FF8C00">''src/''</span> subdirectory, so let's keep to out word and create it. Then, add a new <span style="color:#1E90FF">''main.cpp''</span> file inside and paste the following lines:

<syntaxhighlight lang=CPP>// Original code by Geoffrey Biggs, taken from the PCL tutorial in
// http://pointclouds.org/documentation/tutorials/pcl_visualizer.php

// Simple OpenNI viewer that also allows to write the current scene to a .pcd
// when pressing SPACE.

#include <pcl/io/openni_grabber.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/console/parse.h>

#include <iostream>

using namespace std;
using namespace pcl;

PointCloud<PointXYZRGBA>::Ptr cloudptr(new PointCloud<PointXYZRGBA>); // A cloud that will store color info.
PointCloud<PointXYZ>::Ptr fallbackCloud(new PointCloud<PointXYZ>); // A fallback cloud with just depth data.
boost::shared_ptr<visualization::CloudViewer> viewer; // Point cloud viewer object.
Grabber* openniGrabber; // OpenNI grabber that takes data from the device.
unsigned int filesSaved = 0; // For the numbering of the clouds saved to disk.
bool saveCloud(false), noColor(false); // Program control.

void
printUsage(const char* programName)
{
cout << "Usage: " << programName << " [options]"
<< endl
<< endl
<< "Options:\n"
<< endl
<< "\t<none> start capturing from an OpenNI device.\n"
<< "\t-v FILE visualize the given .pcd file.\n"
<< "\t-h shows this help.\n";
}

// This function is called every time the device has new data.
void
grabberCallback(const PointCloud<PointXYZRGBA>::ConstPtr& cloud)
{
if (! viewer->wasStopped())
viewer->showCloud(cloud);

if (saveCloud)
{
stringstream stream;
stream << "inputCloud" << filesSaved << ".pcd";
string filename = stream.str();
if (io::savePCDFile(filename, *cloud, true) == 0)
{
filesSaved++;
cout << "Saved " << filename << "." << endl;
}
else PCL_ERROR("Problem saving %s.\n", filename.c_str());

saveCloud = false;
}
}

// For detecting when SPACE is pressed.
void
keyboardEventOccurred(const visualization::KeyboardEvent& event,
void* nothing)
{
if (event.getKeySym() == "space" && event.keyDown())
saveCloud = true;
}

// Creates, initializes and returns a new viewer.
boost::shared_ptr<visualization::CloudViewer>
createViewer()
{
boost::shared_ptr<visualization::CloudViewer> v
(new visualization::CloudViewer("OpenNI viewer"));
v->registerKeyboardCallback(keyboardEventOccurred);

return (v);
}

int
main(int argc, char** argv)
{
if (console::find_argument(argc, argv, "-h") >= 0)
{
printUsage(argv[0]);
return -1;
}

bool justVisualize(false);
string filename;
if (console::find_argument(argc, argv, "-v") >= 0)
{
if (argc != 3)
{
printUsage(argv[0]);
return -1;
}

filename = argv[2];
justVisualize = true;
}
else if (argc != 1)
{
printUsage(argv[0]);
return -1;
}

// First mode, open and show a cloud from disk.
if (justVisualize)
{
// Try with color information...
try
{
io::loadPCDFile<PointXYZRGBA>(filename.c_str(), *cloudptr);
}
catch (PCLException e1)
{
try
{
// ...and if it fails, fall back to just depth.
io::loadPCDFile<PointXYZ>(filename.c_str(), *fallbackCloud);
}
catch (PCLException e2)
{
return -1;
}

noColor = true;
}

cout << "Loaded " << filename << "." << endl;
if (noColor)
cout << "This cloud has no RGBA color information present." << endl;
else cout << "This cloud has RGBA color information present." << endl;
}
// Second mode, start fetching and displaying frames from the device.
else
{
openniGrabber = new OpenNIGrabber();
if (openniGrabber == 0)
return -1;
boost::function<void (const PointCloud<PointXYZRGBA>::ConstPtr&)> f =
boost::bind(&grabberCallback, _1);
openniGrabber->registerCallback(f);
}

viewer = createViewer();

if (justVisualize)
{
if (noColor)
viewer->showCloud(fallbackCloud);
else viewer->showCloud(cloudptr);
}
else openniGrabber->start();

// Main loop.
while (! viewer->wasStopped())
boost::this_thread::sleep(boost::posix_time::seconds(1));

if (! justVisualize)
openniGrabber->stop();
}</syntaxhighlight>

Save and close.

==Compiling==

Follow the same steps you used to build PCL. That is, create a new <span style="color:#FF8C00">''build/''</span> subdirectory next to the <span style="color:#FF8C00">''src/''</span> one. Open a terminal there and issue:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..
make</syntaxhighlight>

==Executing==

Still from the same terminal, run the compiled example program:

<syntaxhighlight lang=Bash>./openniViewer</syntaxhighlight>

After some seconds, the main window will appear and the application will start grabbing frames from the device. You can inspect the current point cloud using the mouse, holding the left button to rotate, the right one (or the mouse wheel) to zoom, and the middle one to pan the camera around. At first, you may see only a black screen, or some big colored axes, but no cloud. Try zooming out, to see the whole scene. Another useful key is <span style="color:#228B22">'''R'''</span>, which will reset the camera parameters when pressed. Use it whenever you notice that zooming has gotten slow after some camera movement, or if you still can not see the cloud. See the [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer] tutorial for additional controls and features.

Whenever you feel ready, press the <span style="color:#228B22">'''SPACE'''</span> key. The program will pause for a fraction of a second and the output <span style="color:#FF1493">''"Saved inputCloud0.pcd."''</span> will appear on the console. Check the current folder to see that file <span style="color:#1E90FF">''inputCloud0.pcd''</span> has indeed been written. You can now close the program with <span style="color:#228B22">'''Q'''</span> or <span style="color:#228B22">'''Alt+F4'''</span>.

Next, run it again giving the following parameter:

<syntaxhighlight lang=Bash>./openniViewer -v inputCloud0.pcd</syntaxhighlight>

This will tell the program not to take data from the device, but from the saved point cloud file instead. After it loads, you will realize that you are presented the same scene you saved to disk.

<span style="color:#606060">'''''NOTE: PCD data is saved relative to the sensor. No matter how much you have manipulated the view, it will reset to default when you load the file.'''''</span>

=Conclusion=

At this point, your Kinect device should be working and getting depth data for you. There is a collection of excellent [http://pointclouds.org/documentation/tutorials/ tutorials] for PCL in the official webpage. I encourage you to finish them all before proceeding your experiments with the Kinect sensor. You can also find a good introduction/tutorial at the PCL library [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Introduction.pdf here].

If you use an ASUS Xtion PRO device instead, you should have gotten everything to work without problems or additional steps (except maybe for [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO | this one]]).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-05T16:51:09Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [https://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object. Another important note: the function <span style="color:#FF1493">"addCoordinateSystem()"</span> with those parameters seen above is marked as deprecated in new versions of PCL. From these on you will have to use something like this:

<syntaxhighlight lang=CPP>viewer.addCoordinateSystem(1.0, "reference", 0);</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [https://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [https://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [https://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [https://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [https://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [https://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://en.wikipedia.org/wiki/CUDA CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [https://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 4: 3D object recognition (descriptors)

2015-11-05T16:42:05Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

It is time to learn the basics of one of the most interesting applications of point cloud processing: 3D object recognition. Akin to 2D recognition, this technique relies on finding good keypoints (characteristic points) in the cloud, and matching them to a set of previously saved ones. But 3D has several advantages over 2D: namely, we will be able to estimate with decent accuracy the exact position and orientation of the object, relative to the sensor. Also, 3D object recognition tends to be more robust to clutter (crowded scenes where objects in the front occluding objects in the background). And finally, having information about the object's shape will help with collision avoidance or grasping operations.

In this first tutorial we will see what descriptors are, how many types are there available in PCL, and how to compute them.

=Overview=

The basis of 3D object recognition is to find a set of correspondences between two different clouds, one of them containing the object we are looking for. In order to do this, we need a way to compare points in an unambiguous manner. Until now, we have worked with points that store the XYZ coordinates, the RGB color... but none of those properties are unique enough. In two sequential scans, two points could share the same coordinates despite belonging to different surfaces, and using the color information takes us back to 2D recognition, will all the lightning related problems.

In a previous tutorial, we talked about [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Feature_estimation | features]], before introducing the normals. Normals are an example of feature, because they encode information about the vicinity of the point. That is, the neighboring points are taken into account when computing it, giving us an idea of how the surrounding surface is. But this is not enough. For a feature to be optimal, it must meet the following criteria:

# '''It must be robust to transformations''': rigid transformations (the ones that do not change the distance between points) like translations and rotations must not affect the feature. Even if we play with the cloud a bit beforehand, there should be no difference.
# '''It must be robust to noise''': measurement errors that cause noise should not change the feature estimation much.
# '''It must be resolution invariant''': if sampled with different density (like after performing downsampling), the result must be identical or similar.

This is where descriptors come in. They are more complex (and precise) signatures of a point, that encode a lot of information about the surrounding geometry. The purpose is to unequivocally identify a point across multiple point clouds, no matter the noise, resolution or transformations. Also, some of them capture additional data about the object they belong to, like the viewpoint (that lets us retrieve the pose).

[[Image:Correspondence.jpg|thumb|center|600px|Finding correspondences between point features of two clouds (image from http://pointclouds.org/).]]

There are many 3D descriptors implemented into PCL. Each one has its own method for computing unique values for a point. Some use the difference between the angles of the normals of the point and its neighbors, for example. Others use the distances between the points. Because of this, some are inherently better or worse for certain purposes. A given descriptor may be scale invariant, and another one may be better with occlusions and partial views of objects. Which one you choose depends on what you want to do.

After calculating the necessary values, an additional step is performed to reduce the descriptor size: the result is binned into an [https://en.wikipedia.org/wiki/Histogram histogram]. To do this, the value range of each variable that makes up the descriptor is divided into ''n'' subdivisions, and the number of occurrences in each one is counted. Try to imagine a descriptor that computes a single variable, that ranges from 1 to 100. We choose to create 10 bins for it, so the first bin would gather all occurrences between 1 and 10, the second from 11 to 20, and so on. We look at the value of the variable for the first point-neighbor pair, and it is 27, so we increment the value of the third bin by 1. We keep doing this until we get a final histogram for that keypoint. The bin size must be carefully chosen depending on how descriptive that variable is (the variables do not have to share the same number of bins, and also the bins do not have to be of the same size; if for example most values from the previous example fell in the 50-100 range then it would be sensible to have more bins of smaller size in that range).

Descriptors can be classified in two main categories: global and local. The process for computing and using each one (recognition pipeline) is different, so each will be explained in its own section in this article.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/how_features_work.php How 3D Features work in PCL]
** [https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features Overview and Comparison of Features]
** [http://www.pointclouds.org/assets/icra2013/pcl_features_icra13.pdf How does a good feature look like?]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

==Table==

The following table will give you a hint of how many descriptors are there in PCL, and some of their features:

{| class="wikitable sortable" style="margin: auto; text-align:center" cellpadding="15"
|+ 3D descriptors
|- style="vertical-align:middle;"
! scope="col"| Name
! scope="col"| Type
! scope="col"| Size<sup>†</sup>
! scope="col"| Custom PointType<sup>††</sup>
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#PFH|PFH]] (Point Feature Histogram)
| style="color: sienna;" | Local
| 125
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#FPFH|FPFH]] (Fast Point Feature Histogram)
| style="color: sienna;" | Local
| 33
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RSD|RSD]] (Radius-Based Surface Descriptor)
| style="color: sienna;" | Local
| 289
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#3DSC|3DSC]] (3D Shape Context)
| style="color: sienna;" | Local
| 1980
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#USC|USC]] (Unique Shape Context)
| style="color: sienna;" | Local
| 1960
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#SHOT|SHOT]] (Signatures of Histograms of Orientations)
| style="color: sienna;" | Local
| 352
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Spin_image|Spin image]]
| style="color: sienna;" | Local
| 153*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RIFT|RIFT]] (Rotation-Invariant Feature Transform)
| style="color: sienna;" | Local
| 32*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#NARF|NARF]] (Normal Aligned Radial Feature)
| style="color: sienna;" | Local
| 36
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RoPS|RoPS]] (Rotational Projection Statistics)
| style="color: sienna;" | Local
| 135*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#VFH|VFH]] (Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#CVFH|CVFH]] (Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#OUR-CVFH|OUR-CVFH]] (Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#ESF|ESF]] (Ensemble of Shape Functions)
| style="color: darkcyan;" | Global
| 640
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GFPFH|GFPFH]] (Global Fast Point Feature Histogram)
| style="color: darkcyan;" | Global
| 16
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GRSD|GRSD]] (Global Radius-Based Surface Descriptor)
| style="color: darkcyan;" | Global
| 21
| style="color: green;" | Yes
|}

† Values marked with an asterisk (*) indicate that the descriptor's size depends on some parameter(s), and the one given is for the default values.

†† Descriptors without their own custom PointType use the generic <span style="color:#FF1493">"pcl::Histogram<>"</span> type. See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|Saving and loading]].

Optionally, you can download a document with a less simple version of the table. Page format is A4, landscape:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Downloads''':
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.odt Table as original ODT] (uses font embedding, requires LibreOffice 4)
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.pdf Table as PDF]
</div>

=Local descriptors=

Local descriptors are computed for individual points that we give as input. They have no notion of what an object is, they just describe how the local geometry is around that point. Usually, it is your task to choose which points you want a descriptor to be computed for: the ''keypoints''. Most of the time, you can get away by just performing a downsampling and choosing all remaining points, but keypoint detectors are available, like the one used for [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF]], or [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#ISS|ISS]].

Local descriptors are used for object recognition and [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration | registration]]. Now we will see which ones are implemented into PCL.

==PFH==

PFH stands for Point Feature Histogram. It is one of the most important descriptors offered by PCL and the basis of others such as FPFH. The PFH tries to capture information of the geometry surrounding the point by analyzing the difference between the directions of the normals in the vicinity (and because of this, an imprecise normal estimation may produce low-quality descriptors).

First, the algorithm pairs all points in the vicinity (not just the chosen keypoint with its neighbors, but also the neighbors with themselves). Then, for each pair, a [https://en.wikipedia.org/wiki/Darboux_frame fixed coordinate frame] is computed from their normals. With this frame, the difference between the normals can be encoded with 3 angular variables. These variables, together with the euclidean distance between the points, are saved, and then binned to an histogram when all pairs have been computed. The final descriptor is the concatenation of the histograms of each variable (4 in total).

<center><gallery widths=300px>
File:PFH_neighbors.png | Point pairs established when computing the PFH for a point (image from http://pointclouds.org).
File:PFH_frame.png | Fixed coordinate frame and angular features computed for one of the pairs (image from http://pointclouds.org).
</gallery></center>

Computing descriptors in PCL is very easy, and the PFH is not an exception:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/pfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the PFH descriptors for each point.
pcl::PointCloud<pcl::PFHSignature125>::Ptr descriptors(new pcl::PointCloud<pcl::PFHSignature125>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// PFH estimation object.
pcl::PFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::PFHSignature125> pfh;
pfh.setInputCloud(cloud);
pfh.setInputNormals(normals);
pfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
pfh.setRadiusSearch(0.05);

pfh.compute(*descriptors);
}</syntaxhighlight>

As you can see, PCL uses the <span style="color:#FF1493">"PFHSignature125"</span> type to save the descriptor to. This means that the descriptor's size is 125 (the dimensionality of the feature vector). Dividing a feature in ''D'' dimensional space in ''B'' divisions requires a total of B<sup>D</sup> bins. The original proposal makes use of the distance between the points, but the implementation of PCL does not, as it was not considered discriminative enough (specially in 2.5D scans, where the distance between points increases the further away from the sensor). Hence, the remaining features (with one dimension each) can be divided in 5 divisions, resulting in a 125-dimensional (5<sup>3</sup>) vector.

The final object that stores the computed descriptors can be handled like a normal cloud (even saved to, or read from, disk), and it has the same number of "points" than the original one. The <span style="color:#FF1493">"PFHSignature125"</span> object at position 0, for example, stores the PFH descriptor for the <span style="color:#FF1493">"PointXYZ"</span> point at the same position in the cloud.

For additional details about the descriptor, check the original publications listed below, or PCL's tutorial.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': PFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pfh_estimation.php Point Feature Histograms (PFH) descriptors]
* '''Publications''':
** [http://ias.in.tum.de/_media/spezial/bib/rusu08ias.pdf Persistent Point Feature Histograms for 3D Point Clouds] (Radu Bogdan Rusu et al., 2008)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.4, page 50)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_p_f_h_estimation.html pcl::PFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_PFH.tar.gz Download]
</div>

===FPFH===

PFH gives accurate results, but it has a drawback: it is too computationally expensive to perform at real time. For a cloud of ''n'' keypoints with ''k'' neighbors considered, it has a [https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations complexity] of ''O(nk<sup>2</sup>)''. Because of this, a derived descriptor was created, named FPFH (Fast Point Feature Histogram).

The FPFH considers only the direct connections between the current keypoint and its neighbors, removing additional links between neighbors. This takes the complexity down to ''O(nk)''. Because of this, the resulting histogram is referred to as SPFH (Simplified Point Feature Histogram). The reference frame and the angular variables are computed as always.

[[Image:FPFH_neighbors.png|thumb|center|400px|Point pairs established when computing the FPFH for a point (image from http://pointclouds.org).]]

To account for the loss of these extra connections, an additional step takes place after all histograms have been computed: the SPFHs of a point's neighbors are "merged" with its own, weighted according to the distance. This has the effect of giving a point surface information of points as far away as 2 times the radius used. Finally, the 3 histograms (distance is not used) are concatenated to compose the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/fpfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the FPFH descriptors for each point.
pcl::PointCloud<pcl::FPFHSignature33>::Ptr descriptors(new pcl::PointCloud<pcl::FPFHSignature33>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// FPFH estimation object.
pcl::FPFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::FPFHSignature33> fpfh;
fpfh.setInputCloud(cloud);
fpfh.setInputNormals(normals);
fpfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
fpfh.setRadiusSearch(0.05);

fpfh.compute(*descriptors);
}</syntaxhighlight>

An additional implementation of the FPFH estimation that takes advantage of multithreaded optimizations (with the [https://en.wikipedia.org/wiki/OpenMP OpenMP API]) is available in the class <span style="color:#FF1493">"FPFHEstimationOMP"</span>. Its interface is identical to the standard unoptimized implementation. Using it will result in a big performance boost on multi-core systems, meaning faster computation times. Remember to include the header <span style="color:#FF1493">"fpfh_omp.h"</span> instead.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': FPFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/fpfh_estimation.php Fast Point Feature Histograms (FPFH) descriptors]
* '''Publications''':
** [http://files.rbrusu.com/publications/Rusu09ICRA.pdf Fast Point Feature Histograms (FPFH) for 3D Registration] (Radu Bogdan Rusu et al., 2009)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.5, page 57)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation.html pcl::FPFHEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation_o_m_p.html pcl::FPFHEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_FPFH.tar.gz Download]
</div>

==RSD==

The Radius-Based Surface Descriptor encodes the radial relationship of the point and its neighborhood. For every pair of the keypoint with a neighbor, the algorithm computes the distance between them, and the difference between their normals. Then, by assuming that both points lie on the surface of a sphere, said sphere is found by fitting not only the points, but also the normals (otherwise, there would be infinite possible spheres). Finally, from all the point-neighbor spheres, only the ones with the maximum and minimum radii are kept and saved to the descriptor of that point.

As you may have deduced already, when two points lie on a flat surface, the sphere radius will be infinite. If, on the other hand, they lie on the curved face of a cylinder, the radius will be more or less the same as that of the cylinder. This allows us to tell objects apart with RSD. The algorithm takes a parameter that sets the maximum radius at which the points will be considered to be part of a plane.

<center><gallery widths=300px>
File:RSD_sphere.png | Sphere that fits two points with their normals (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
File:RSD_values.png | RSD values for a cloud; points on a flat surface have maximum values (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
</gallery></center>

This is the code for compiling the RSD descriptor:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/rsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RSD descriptors for each point.
pcl::PointCloud<pcl::PrincipalRadiiRSD>::Ptr descriptors(new pcl::PointCloud<pcl::PrincipalRadiiRSD>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// RSD estimation object.
pcl::RSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::PrincipalRadiiRSD> rsd;
rsd.setInputCloud(cloud);
rsd.setInputNormals(normals);
rsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
rsd.setRadiusSearch(0.05);
// Plane radius. Any radius larger than this is considered infinite (a plane).
rsd.setPlaneRadius(0.1);
// Do we want to save the full distance-angle histograms?
rsd.setSaveHistograms(false);

rsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

Optionally, you can use the <span style="color:#FF1493">"setSaveHistograms()"</span> function to enable the saving of the full distance-angle histograms, and then use <span style="color:#FF1493">"getHistograms()"</span> to retrieve them.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Maximum Radius, [Subdivisions], [Save full histograms]
* '''Output''': RSD descriptors
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.391.7398&rep=rep1&type=pdf General 3D Modelling of Novel Objects from a Single View] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_s_d_estimation.html pcl::RSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RSD.tar.gz Download]
</div>

==3DSC==

The 3D Shape Context is a descriptor that extends its existing 2D counterpart to the third dimension. It works by creating a support structure (a sphere, to be precise) centered at the point we are computing the descriptor for, with the given search radius. The "north pole" of that sphere (the notion of "up") is pointed to match the normal at that point. Then, the sphere is divided in 3D regions or bins. In the first 2 coordinates (azimuth and elevation) the divisions are equally spaced, but in the third (the radial dimension), divisions are logarithmically spaced so they are smaller towards the center. A minimum radius can be specified to prevent very small bins, that would be too sensitive to small changes in the surface.

[[Image:3DSC_support_structure.png|thumb|center|200px|Support structure to compute the 3DSC for a point (image from [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf original paper]).]]

For each bin, a weighted count is accumulated for every neighboring point that lies within. The weight depends on the volume of the bin and the local point density (number of points around the current neighbor). This gives the descriptor some degree of resolution invariance.

We have mentioned that the sphere is given the direction of the normal. This still leaves one degree of freedom (only two axes have been locked, the azimuth remains free). Because of this, the descriptor so far does not cope with rotation. To overcome this (so the same point in two different clouds has the same value), the support sphere is rotated around the normal ''N'' times (a number of degrees that corresponds with the divisions in the azimuth) and the process is repeated for each, giving a total of ''N'' descriptors for that point.

You can compute the 3DSC descriptor the following way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/3dsc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the 3DSC descriptors for each point.
pcl::PointCloud<pcl::ShapeContext1980>::Ptr descriptors(new pcl::PointCloud<pcl::ShapeContext1980>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// 3DSC estimation object.
pcl::ShapeContext3DEstimation<pcl::PointXYZ, pcl::Normal, pcl::ShapeContext1980> sc3d;
sc3d.setInputCloud(cloud);
sc3d.setInputNormals(normals);
sc3d.setSearchMethod(kdtree);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
sc3d.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
sc3d.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
sc3d.setPointDensityRadius(0.05 / 5.0);

sc3d.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Minimal radius, Point density radius
* '''Output''': 3DSC descriptors
* '''Publication''':
** [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf Recognizing Objects in Range Data Using Regional Point Descriptors] (Andrea Frome et al., 2004)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_shape_context3_d_estimation.html pcl::ShapeContext3DEstimation]
* [http://robotica.unileon.es/~victorm/PCL_3DSC.tar.gz Download]
</div>

===USC===

The Unique Shape Context descriptor extends the 3DSC by defining a local reference frame, in order to provide an unique orientation for each point. This not only improves the accuracy of the descriptor, it also reduces its size, as computing multiple descriptors to account for orientation is no longer necessary.

You can check the second publication listed below to learn more about how the LRF is computed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/usc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the USC descriptors for each point.
pcl::PointCloud<pcl::UniqueShapeContext1960>::Ptr descriptors(new pcl::PointCloud<pcl::UniqueShapeContext1960>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// USC estimation object.
pcl::UniqueShapeContext<pcl::PointXYZ, pcl::UniqueShapeContext1960, pcl::ReferenceFrame> usc;
usc.setInputCloud(cloud);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
usc.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
usc.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
usc.setPointDensityRadius(0.05 / 5.0);
// Set the radius to compute the Local Reference Frame.
usc.setLocalRadius(0.05);

usc.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk). For 1.7 and below, change UniqueShapeContext1960 to ShapeContext1980, and edit CMakeLists.txt.'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Minimal radius, Point density radius, Local radius
* '''Output''': USC descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/3dor10.pdf Unique Shape Context for 3D Data Description] (Federico Tombari et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_unique_shape_context.html pcl::UniqueShapeContext]
* [http://robotica.unileon.es/~victorm/PCL_USC.tar.gz Download]
</div>

==SHOT==

SHOT stands for Signature of Histograms of Orientations. Like 3DSC, it encodes information about the topology (surface) withing a spherical support structure. This sphere is divided in 32 bins or volumes, with 8 divisions along the azimuth, 2 along the elevation, and 2 along the radius. For every volume, a one-dimensional local histogram is computed. The variable chosen is the angle between the normal of the keypoint and the current point within that volume (to be precise, the cosine, which was found to be better suitable).

[[Image:SHOT_support_structure.png|thumb|center|200px|Support structure to compute SHOT. Only 4 azimuth divisions are shown for clarity (image from [http://vision.deis.unibo.it/fede/papers/eccv10.pdf original paper]).]]

When all local histograms have been computed, they are stitched together in a final descriptor. Like the USC descriptor, SHOT makes use of a local reference frame, making it rotation invariant. It is also robust to noise and clutter.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the SHOT descriptors for each point.
pcl::PointCloud<pcl::SHOT352>::Ptr descriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// SHOT estimation object.
pcl::SHOTEstimation<pcl::PointXYZ, pcl::Normal, pcl::SHOT352> shot;
shot.setInputCloud(cloud);
shot.setInputNormals(normals);
// The radius that defines which of the keypoint's neighbors are described.
// If too large, there may be clutter, and if too small, not enough points may be found.
shot.setRadiusSearch(0.02);

shot.compute(*descriptors);
}</syntaxhighlight>

Like with FPFH, a multithreading-optimized variant is available with <span style="color:#FF1493">"SHOTEstimationOMP"</span>, that makes use of OpenMP. You need to include the header <span style="color:#FF1493">"shot_omp.h"</span>. Also, another variant that uses the texture for matching is available, <span style="color:#FF1493">"SHOTColorEstimation"</span>, with an optimized version too (see the second publication for more details). It outputs a <span style="color:#FF1493">"SHOT1344"</span> descriptor.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius
* '''Output''': SHOT descriptors
* '''Publications''':
** [http://vision.deis.unibo.it/fede/papers/eccv10.pdf Unique Signatures of Histograms for Local Surface Description] (Federico Tombari et al., 2010)
** [http://www.vision.deis.unibo.it/fede/papers/icip11.pdf A Combined Texture-Shaped Descriptor for Enhanced 3D Feature Matching] (Federico Tombari et al., 2011)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation.html pcl::SHOTEstimation],
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation.html pcl::SHOTColorEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation_o_m_p.html pcl::SHOTEstimationOMP]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation_o_m_p.html pcl::SHOTColorEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_SHOT.tar.gz Download]
</div>

==Spin image==

The Spin Image (SI) is the oldest descriptor we are going to see here. It has been around since 1997, but it still sees some use for certain applications. It was originally designed to describe surfaces made by vertices, edges and polygons, but it has been since adapted for point clouds. The descriptor is unlike all others in that the output resembles an image that can be compared with another with the usual means.

The support structure used is a cylinder, centered at the point, with a given radius and height, and aligned with the normal. This cylinder is divided radially and vertically into volumes. For each one, the number of neighbors lying inside is added up, eventually producing a descriptor. Weighting and interpolation are used to improve the result. The final descriptor can be seen as a grayscale image where dark areas correspond to volumes with higher point density.

[[Image:Spin images.png|thumb|center|400px|Spin images computed for 3 points of a model (image from [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf original thesis]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/spin_image.h>

// A handy typedef.
typedef pcl::Histogram<153> SpinImage;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the spin image for each point.
pcl::PointCloud<SpinImage>::Ptr descriptors(new pcl::PointCloud<SpinImage>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Spin image estimation object.
pcl::SpinImageEstimation<pcl::PointXYZ, pcl::Normal, SpinImage> si;
si.setInputCloud(cloud);
si.setInputNormals(normals);
// Radius of the support cylinder.
si.setRadiusSearch(0.02);
// Set the resolution of the spin image (the number of bins along one dimension).
// Note: you must change the output histogram size to reflect this.
si.setImageWidth(8);

si.compute(*descriptors);
}</syntaxhighlight>

The Spin Image estimation object provides more methods for tuning the estimation, so checking the API is recommended.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius, Image resolution
* '''Output''': Spin images
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf Spin-Images: A Representation for 3-D Surface Matching] (Andrew Edie Johnson, 1997)
** [http://www.cs.jhu.edu/~misha/Papers/Johnson99.pdf Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes] (Andrew Edie Johnson and Martial Hebert, 1999)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_spin_image_estimation.html pcl::SpinImageEstimation]
* [http://robotica.unileon.es/~victorm/PCL_spin_image.tar.gz Download]
</div>

==RIFT==

The Rotation-Invariant Feature Transform, like the spin image, takes some concepts from 2D features, in this case from the Scale-Invariant Feature Transform ([https://en.wikipedia.org/wiki/Scale-invariant_feature_transform SIFT]). It is the only descriptor seen here that requires intensity information in order to compute it (it can be obtained from the RGB color values). This means, of course, that you will not be able to use RIFT with standard XYZ clouds, you also need the texture.

In the first step, a circular patch (with the given radius) is fitted on the surface the point lies on. This patch is divided into concentric rings, according to the chosen distance bin size. Then, an histogram is populated with all the point's neighbors lying inside a sphere centered at that point and with the mentioned radius. The distance and the orientation of the intensity gradient at each point are considered. To make it rotation invariant, the angle between the gradient orientation and the vector pointing outward from the center of the patch is measured.

[[Image:RIFT.png|thumb|center|600px|RIFT feature values at 3 different locations in the descriptor (image from [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf original paper]).]]

The authors' original implementation uses 4 rings and 8 histogram orientations, which produce a descriptor of size 32. RIFT is not robust to texture flipping, though this was never considered a big issue.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/intensity_gradient.h>
#include <pcl/features/rift.h>

// A handy typedef.
typedef pcl::Histogram<32> RIFT32;

int
main(int argc, char** argv)
{
// Object for storing the point cloud with color information.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloudColor(new pcl::PointCloud<pcl::PointXYZRGB>);
// Object for storing the point cloud with intensity value.
pcl::PointCloud<pcl::PointXYZI>::Ptr cloudIntensity(new pcl::PointCloud<pcl::PointXYZI>);
// Object for storing the intensity gradients.
pcl::PointCloud<pcl::IntensityGradient>::Ptr gradients(new pcl::PointCloud<pcl::IntensityGradient>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RIFT descriptor for each point.
pcl::PointCloud<RIFT32>::Ptr descriptors(new pcl::PointCloud<RIFT32>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloudColor) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Convert the RGB to intensity.
pcl::PointCloudXYZRGBtoXYZI(*cloudColor, *cloudIntensity);

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZI, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudIntensity);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZI>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZI>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Compute the intensity gradients.
pcl::IntensityGradientEstimation < pcl::PointXYZI, pcl::Normal, pcl::IntensityGradient,
pcl::common::IntensityFieldAccessor<pcl::PointXYZI> > ge;
ge.setInputCloud(cloudIntensity);
ge.setInputNormals(normals);
ge.setRadiusSearch(0.03);
ge.compute(*gradients);

// RIFT estimation object.
pcl::RIFTEstimation<pcl::PointXYZI, pcl::IntensityGradient, RIFT32> rift;
rift.setInputCloud(cloudIntensity);
rift.setSearchMethod(kdtree);
// Set the intensity gradients to use.
rift.setInputGradient(gradients);
// Radius, to get all neighbors within.
rift.setRadiusSearch(0.02);
// Set the number of bins to use in the distance dimension.
rift.setNrDistanceBins(4);
// Set the number of bins to use in the gradient orientation dimension.
rift.setNrGradientBins(8);
// Note: you must change the output histogram size to reflect the previous values.

rift.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Intensity gradients, Radius, Distance bins, Gradient bins
* '''Output''': RIFT descriptors
* '''Publication''':
** [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf A Sparse Texture Representation Using Local Affine Regions] (Svetlana Lazebnik et al., 2005)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_r_i_f_t_estimation.html pcl::RIFTEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_intensity_gradient_estimation.html pcl::IntensityGradientEstimation]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_intensity_gradient.html pcl::IntensityGradient]
* [http://robotica.unileon.es/~victorm/PCL_RIFT.tar.gz Download]
</div>

==NARF==

The Normal Aligned Radial Feature is the only descriptor here that does not take a point cloud as input. Instead, it works with range images. A range image is a common RGB image in which the distance to the point that corresponds to a certain pixel is encoded as a color value in the [https://en.wikipedia.org/wiki/Visible_spectrum visible light spectrum]: the points that are closer to the camera would be violet, while the points near the maximum sensor range would be red.

NARF also requires us to find suitable keypoints to compute the descriptor for. NARF keypoints are located near an object's corners, and this also requires to find the borders (transitions from foreground to background), which are trivial to find with a range image. Because of this lengthy pipeline, I will describe the whole process in different sections.

===Obtaining a range image===

Because we always work with point clouds, I will now explain how you can convert one into a range image, in order to use it for the NARF descriptor. PCL provides a couple of handy classes to perform the conversion, given that you fill the camera data correctly.

A range image can be created in two ways. First, we can use spherical projection, which would give us an image similar to the ones produced by a LIDAR sensor. Second, we can use planar projection, which is better suitable for camera-like sensors as the Kinect or the Xtion, and will not have the characteristic distortion of the first one.

====Spherical projection====

The following code will take a point cloud and create a range image from it, using spherical projection:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the range image object:

// Angular resolution is the angular distance between pixels.
// Kinect: 57° horizontal FOV, 43° vertical FOV, 640x480 (chosen here).
// Xtion: 58° horizontal FOV, 45° vertical FOV, 640x480.
float angularResolutionX = (float)(57.0f / 640.0f * (M_PI / 180.0f));
float angularResolutionY = (float)(43.0f / 480.0f * (M_PI / 180.0f));
// Maximum horizontal and vertical angles. For example, for a full panoramic scan,
// the first would be 360º. Choosing values that adjust to the real sensor will
// decrease the time it takes, but don't worry. If the values are bigger than
// the real ones, the image will be automatically cropped to discard empty zones.
float maxAngleX = (float)(60.0f * (M_PI / 180.0f));
float maxAngleY = (float)(50.0f * (M_PI / 180.0f));
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;
// Border size. If greater than 0, a border of "unobserved" points will be left
// in the image when it is cropped.
int borderSize = 1;

// Range image object.
pcl::RangeImage rangeImage;
rangeImage.createFromPointCloud(*cloud, angularResolutionX, angularResolutionY,
maxAngleX, maxAngleY, sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange, borderSize);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Range image");
viewer.showRangeImage(rangeImage);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

Here you can see an example of the output range image:

[[Image:Range_image_spherical.png|thumb|center|400px|Range image of a point cloud, using spherical projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Angular resolution, Maximum angle, Sensor pose, Coordinate frame, Noise level, Maximum range, Border size
* '''Output''': Range image (spherical projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image.html pcl::RangeImage]
* [http://robotica.unileon.es/~victorm/PCL_range_image_spherical.tar.gz Download]
</div>

====Planar projection====

As mentioned, planar projection will give better results with clouds taken from a depth camera:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the planar range image object:

// Image size. Both Kinect and Xtion work at 640x480.
int imageSizeX = 640;
int imageSizeY = 480;
// Center of projection. here, we choose the middle of the image.
float centerX = 640.0f / 2.0f;
float centerY = 480.0f / 2.0f;
// Focal length. The value seen here has been taken from the original depth images.
// It is safe to use the same value vertically and horizontally.
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;

// Planar range image object.
pcl::RangeImagePlanar rangeImagePlanar;
rangeImagePlanar.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Planar range image");
viewer.showRangeImage(rangeImagePlanar);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_planar.png|thumb|center|400px|Range image of a point cloud, using planar projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Image size, Projection center, Focal length, Sensor pose, Coordinate frame, Noise level, Maximum range
* '''Output''': Range image (planar projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_planar.html pcl::RangeImagePlanar]
* [http://robotica.unileon.es/~victorm/PCL_range_image_planar.tar.gz Download]
</div>

If you prefer to do the conversion in real time while you inspect the cloud, PCL ships with an [https://github.com/PointCloudLibrary/pcl/tree/master/doc/tutorials/content/sources/openni_range_image_visualization example] that fetches an <span style="color:#FF1493">"openni_wrapper::DepthImage"</span> from an OpenNI device and creates the range image from it. You can adapt the code of the [[PCL/OpenNI_tutorial_1:_Installing_and_testing#Testing | example]] example from tutorial 1 to save it to disk with the function [http://docs.pointclouds.org/trunk/group__io.html#ga7291a029cdcde32ca3639d07dc6491b9 pcl::io::saveRangeImagePlanarFilePNG()].

===Extracting borders===

NARF keypoints are located near the edges of objects in the range image, so in order to find them, we first have to extract the borders. A border is defined as an abrupt change from foreground to background. In a range image, this can be easily seen because there is a "jump" in the depth value of two adjacent pixels.

[[Image:Range_image_border_detection.png|thumb|center|400px|Border detection on a range image (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

There are three types of borders. ''Object borders'' consist of the pixels (or points) located on the very edge of an object (the outermost points still belonging to the object). ''Shadow borders'' are points in the background on the edge of occlusions (empty areas in the background due to the objects in front covering them). Notice that, when the cloud is seen from the sensor's viewpoint, object and shadow points will seem adjacent. Finally, ''veil points'' are points interpolated between the previous two which appear in scans taken with LIDAR sensors, so we do not have to worry about them here.

<center><gallery widths=300px>
File:Range_image_border_type.png | Types of borders (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:Range_image_borders_example.png | Example of object and shadow borders on a cloud.
</gallery></center>

The algorithm basically compares a point's depth with the values of its neighbors, and if a big difference is found, we know it is due to a border. Points closer to the sensor will be marked as object borders, and the other ones as shadow borders.

PCL provides a class for extracting borders of a range image:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the borders.
pcl::PointCloud<pcl::BorderDescription>::Ptr borders(new pcl::PointCloud<pcl::BorderDescription>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Border extractor object.
pcl::RangeImageBorderExtractor borderExtractor(&rangeImage);

borderExtractor.compute(*borders);

// Visualize the borders.
pcl::visualization::RangeImageVisualizer* viewer = NULL;
viewer = pcl::visualization::RangeImageVisualizer::getRangeImageBordersWidget(rangeImage,
-std::numeric_limits<float>::infinity(),
std::numeric_limits<float>::infinity(),
false, *borders, "Borders");

while (!viewer->wasStopped())
{
viewer->spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_borders.png|thumb|center|400px|Borders found on the range image.]]

You can use the extractor's <span style="color:#FF1493">"getParameters()"</span> function to get a [http://docs.pointclouds.org/trunk/structpcl_1_1_range_image_border_extractor_1_1_parameters.html pcl::RangeImageBorderExtractor::Parameters] struct with the settings that will be used.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image
* '''Output''': Borders
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/range_image_border_extraction.php How to extract borders from range images]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_border_extractor.html pcl::RangeImageBorderExtractor]
* [http://robotica.unileon.es/~victorm/PCL_range_image_borders.tar.gz Download]
</div>

===Finding keypoints===

Citing the [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original publication]:''

''"We have the following requirements for our interest point extraction procedure:

# ''It must take information about borders and the surface structure into account.''
# ''It must select positions that can be reliably detected even if the object is observed from another perspective.''
# ''The points must be on positions that provide stable areas for normal estimation or the descriptor calculation in general."''

The procedure is the following: for every point in the range image, a score is computed that conveys how much the surface changes in its neighborhood (this is tuned with the ''support size'' σ, which is the diameter of the sphere used to find neighboring points). Also, the dominant direction of this change is computed. Then, this direction is compared with those of the neighbors, trying to find how stable the point is (if the directions are very different, that means the point is not stable, and that the surface around changes a lot). Points that are near the object's corners (but not exactly on the very edge) will be good keypoints, yet stable enough.

<center><gallery widths=300px>
File:NARF_keypoints.png | NARF keypoints (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:NARF_keypoints_support_sizes.png | Interest regions with a support size of 20cm (up) and 1m (down) (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
</gallery></center>

In PCL, NARF keypoints can be found this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

pcl::RangeImageBorderExtractor borderExtractor;
// Keypoint detection object.
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
// The support size influences how big the surface of interest will be,
// when finding keypoints from the border information.
detector.getParameters().support_size = 0.2f;

detector.compute(*keypoints);

// Visualize the keypoints.
pcl::visualization::RangeImageVisualizer viewer("NARF keypoints");
viewer.showRangeImage(rangeImage);
for (size_t i = 0; i < keypoints->points.size(); ++i)
{
viewer.markPoint(keypoints->points[i] % rangeImage.width,
keypoints->points[i] / rangeImage.width,
// Set the color of the pixel to red (the background
// circle is already that color). All other parameters
// are left untouched, check the API for more options.
pcl::visualization::Vector3ub(1.0f, 0.0f, 0.0f));
}

while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_NARF_keypoints.png|thumb|center|400px|NARF keypoints found on the range image.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Border extractor, Support Size
* '''Output''': NARF keypoints
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_keypoint_extraction.php How to extract NARF keypoint from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_keypoint.html pcl::NarfKeypoint]
* [http://robotica.unileon.es/~victorm/PCL_NARF_keypoints.tar.gz Download]
</div>

===Computing the descriptor===

We have created the range image from a point cloud, and we have extracted the borders in order to find good keypoints. Now it is time to compute the NARF descriptor for each keypoint.

The NARF descriptor encodes information about surface changes around a point. First, a local range patch is created around the point. It is like a small range image centered at that point, aligned with the normal (it would seem as if we were looking at the point along the normal). Then, a star pattern with ''n'' beams is overlaid onto the patch, also centered at the point. For every beam, a value is computed, that reflects how much the surface under it changes. The stronger the change is, and the closer to the center it is, the higher the final value will be. The ''n'' resulting values compose the final descriptor.

[[Image:NARF_descriptor.png|thumb|center|600px|Computing the NARF descriptor for a keypoint (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

The descriptor right now is not invariant to rotations around the normal. To achieve this, the whole possible 360 degrees are binned into an histogram. The value of each bin is computed from the descriptor values according to the angle. Then, the bin with the highest value is considered the dominant orientation, and the descriptor is shifted according to it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/features/narf_descriptor.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);
// Object for storing the NARF descriptors.
pcl::PointCloud<pcl::Narf36>::Ptr descriptors(new pcl::PointCloud<pcl::Narf36>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Extract the keypoints.
pcl::RangeImageBorderExtractor borderExtractor;
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
detector.getParameters().support_size = 0.2f;
detector.compute(*keypoints);

// The NARF estimator needs the indices in a vector, not a cloud.
std::vector<int> keypoints2;
keypoints2.resize(keypoints->points.size());
for (unsigned int i = 0; i < keypoints->size(); ++i)
keypoints2[i] = keypoints->points[i];
// NARF estimation object.
pcl::NarfDescriptor narf(&rangeImage, &keypoints2);
// Support size: choose the same value you used for keypoint extraction.
narf.getParameters().support_size = 0.2f;
// If true, the rotation invariant version of NARF will be used. The histogram
// will be shifted according to the dominant orientation to provide robustness to
// rotations around the normal.
narf.getParameters().rotation_invariant = true;

narf.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Key points, Support Size
* '''Output''': NARF descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_feature_extraction.php How to extract NARF Features from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_descriptor.html pcl::NarfDescriptor]
* [http://robotica.unileon.es/~victorm/PCL_NARF_descriptor.tar.gz Download]
</div>

==RoPS==

The Rotational Projection Statistics (RoPS) feature is a bit different from the other descriptors because it works with a triangle mesh, so a previous triangulation step is needed for generating this mesh from the cloud. Apart from that, most concepts are similar.

In order to compute RoPS for a keypoint, the local surface is cropped according to a support radius, so only points and triangles lying inside are taken into account. Then, a local reference frame (LRF) is computed, giving the descriptor its rotational invariance. A coordinate system is created with the point as the origin, and the axes aligned with the LRF. Then, for every axis, several steps are performed.

First, the local surface is rotated around the current axis. The angle is determined by one of the parameters, which sets the number of rotations. Then, all points in the local surface are projected onto the XY, XZ and YZ planes. For each one, statistical information about the distribution of the projected points is computed, and concatenated to form the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/surface/gp3.h>
#include <pcl/features/rops_estimation.h>

// A handy typedef.
typedef pcl::Histogram<135> ROPS135;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing both the points and the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr cloudNormals(new pcl::PointCloud<pcl::PointNormal>);
// Object for storing the ROPS descriptor for each point.
pcl::PointCloud<ROPS135>::Ptr descriptors(new pcl::PointCloud<ROPS135>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Perform triangulation.
pcl::concatenateFields(*cloud, *normals, *cloudNormals);
pcl::search::KdTree<pcl::PointNormal>::Ptr kdtree2(new pcl::search::KdTree<pcl::PointNormal>);
kdtree2->setInputCloud(cloudNormals);
pcl::GreedyProjectionTriangulation<pcl::PointNormal> triangulation;
pcl::PolygonMesh triangles;
triangulation.setSearchRadius(0.025);
triangulation.setMu(2.5);
triangulation.setMaximumNearestNeighbors(100);
triangulation.setMaximumSurfaceAngle(M_PI / 4); // 45 degrees.
triangulation.setNormalConsistency(false);
triangulation.setMinimumAngle(M_PI / 18); // 10 degrees.
triangulation.setMaximumAngle(2 * M_PI / 3); // 120 degrees.
triangulation.setInputCloud(cloudNormals);
triangulation.setSearchMethod(kdtree2);
triangulation.reconstruct(triangles);

// Note: you should only compute descriptors for chosen keypoints. It has
// been omitted here for simplicity.

// RoPs estimation object.
pcl::ROPSEstimation<pcl::PointXYZ, ROPS135> rops;
rops.setInputCloud(cloud);
rops.setSearchMethod(kdtree);
rops.setRadiusSearch(0.03);
rops.setTriangles(triangles.polygons);
// Number of partition bins that is used for distribution matrix calculation.
rops.setNumberOfPartitionBins(5);
// The greater the number of rotations is, the bigger the resulting descriptor.
// Make sure to change the histogram size accordingly.
rops.setNumberOfRotations(3);
// Support radius that is used to crop the local surface of the point.
rops.setSupportRadius(0.025);

rops.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Triangles, Search method, Support radius, Number of rotations, Number of partition bins
* '''Output''': RoPS descriptors
* '''Publication''':
** [http://arxiv.org/pdf/1304.3192v1.pdf Rotational Projection Statistics for 3D Local Surface Description and Object Recognition] (Yulan Guo et al., 2013)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_o_p_s_estimation.html pcl::ROPSEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RoPS.tar.gz Download]
</div>

=Global descriptors=

Global descriptors encode object geometry. They are not computed for individual points, but for a whole cluster that represents an object. Because of this, a preprocessing step ([[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]]) is required, in order to retrieve possible candidates.

Global descriptors are used for object recognition and classification, geometric analysis (object type, shape...), and pose estimation.

You should also know that many local descriptors can also be used as global ones. This can be done with descriptors that use a radius to search for neighbors (as PFH does). The trick is to compute it for one single point in the object cluster, and set the radius to the maximum possible distance between any two points (so all points in the cluster are considered as neighbors).

==VFH==

The Viewpoint Feature Histogram is based on the FPFH. Because the latter is invariant to the object's pose, the authors decided to expand it by including information about the viewpoint. Also, the FPFH is estimated once for the whole cluster, not for every point.

The VFH is made up by two parts: a viewpoint direction component, and an extended FPFH component. To compute the first one, the object's [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]] is found, which is the point that results from averaging the X, Y and Z coordinates of all points. Then, the vector between the viewpoint (the position of the sensor) and this centroid is computed, and normalized. Finally, for all points in the cluster, the angle between this vector and their normal is calculated, and the result is binned into an histogram. The vector is translated to each point when computing the angle because it makes the descriptor scale invariant.

The second component is computed like the FPFH (that results in 3 histograms for the 3 angular features, α, φ and θ), with some differences: it is only computed for the centroid, using the computed viewpoint direction vector as its normal (as the point, obviously, does not have a normal), and setting all the cluster's points as neighbors.

<center><gallery widths=300px>
File:VFH_viewpoint_component.png | Viewpoint component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
File:VFH_extended_FPFH_component.png | Extended FPFH component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
</gallery></center>

The resulting 4 histograms (1 for the viewpoint component, 3 for the extended FPFH component) are concatenated to build the final VFH descriptor. By default, the bins are normalized using the total number of points in the cluster. This makes the VFH descriptor invariant to scale.

[[Image:VFH_histogram.png|thumb|center|400px| VFH histogram (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).]]

The PCL implementation computes an additional fifth histogram with the distances of the cluster points to the centroid (the Shape Distribution Component, SDC), increading the size of the output descriptor from 263 to 308. The SDC is taken from the CVFH descriptor that we will see in the next section, and makes the result more robust.

The VFH of an already clustered object can be computed this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the VFH descriptor.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// VFH estimation object.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
// Optionally, we can normalize the bins of the resulting histogram,
// using the total number of points.
vfh.setNormalizeBins(true);
// Also, we can normalize the SDC with the maximum size found between
// the centroid and any of the cluster's points.
vfh.setNormalizeDistance(false);

vfh.compute(*descriptor);
}</syntaxhighlight>

Because only one VFH descriptor is computed for the whole cluster, the size of the cloud object that stores the result will be 1.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, [Normalize bins], [Normalize SDC]
* '''Output''': VFH descriptor
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_estimation.php Estimating VFH signatures for a set of points]
* '''Publication''':
** [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf Fast 3D Recognition and Pose Using the Viewpoint Feature Histogram] (Radu Bogdan Rusu et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_v_f_h_estimation.html pcl::VFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_VFH.tar.gz Download]
</div>

===CVFH===

The original VFH descriptor is not robust to occlusion or other sensor artifacts, or measurement errors. If the object cluster is missing many points, the resulting computed centroid will differ from the original, altering the final descriptor, and preventing a positive match from being found. Because of that, the Clustered Viewpoint Feature Histogram (CVFH) was introduced.

The idea is very simple: instead of computing a single VFH histogram for the whole cluster, the object is first divided in stable, smooth regions using [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Region_growing | region-growing segmentation]], that enforces several constraints in the distances and differences of normals of the points belonging to every region. Then, a VFH is computed for every region. Thanks to this, an object can be found in a scene, as long as at least one of its regions is fully visible.

<center><gallery widths=300px>
File:CVFH_occlusion.png | Typical occlusion issues in a point cloud (image from original paper).
File:CVFH_regions.png | Object regions computed for the CVFH (image from original paper).
</gallery></center>

Additionally, a Shape Distribution Component (SDC) is also computed and included. It encodes information about the distribution of the points arond the region's centroid, measuring the distances. The SDC allows to differentiate objects with similar characteristics (size and normal distribution), like two planar surfaces from each other.

The authors proposed to discard the histogram normalization step that is performed in VFH. This has the effect of making the descriptor dependant of scale, so an object of a certain size would not match a bigger or smaller copy of itself. It also makes CVFH more robust to occlusion.

CVFH is invariant to the camera roll angle, like most global descriptors. This is so because rotations about that camera axis do not change the observable geometry that descriptors are computed from, limiting the pose estimation to 5 DoF. The use of a Camera Roll Histogram (CRH) has been proposed to overcome this.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CVFH estimation object.
pcl::CVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> cvfh;
cvfh.setInputCloud(object);
cvfh.setInputNormals(normals);
cvfh.setSearchMethod(kdtree);
// Set the maximum allowable deviation of the normals,
// for the region segmentation step.
cvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
// Set the curvature threshold (maximum disparity between curvatures),
// for the region segmentation step.
cvfh.setCurvatureThreshold(1.0);
// Set to true to normalize the bins of the resulting histogram,
// using the total number of points. Note: enabling it will make CVFH
// invariant to scale just like VFH, but the authors encourage the opposite.
cvfh.setNormalizeBins(false);

cvfh.compute(*descriptors);
}</syntaxhighlight>

You can further customize the segmentation step with <span style="color:#FF1493">"setClusterTolerance()"</span> (to set the maximum Euclidean distance between points in the same cluster) and <span style="color:#FF1493">"setMinPoints()"</span>. The size of the output will be equal to the number of regions the object was divided in. Also, check the functions <span style="color:#FF1493">"getCentroidClusters()"</span> and <span style="color:#FF1493">"getCentroidNormalClusters()"</span>, you can use them to get information about the centroids used to compute the different CVFH descriptors.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points]
* '''Output''': CVFH descriptors
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_v_f_h_estimation.html pcl::CVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CVFH.tar.gz Download]
</div>

====OUR-CVFH====

The Oriented, Unique and Repeatable CVFH expands the previous descriptor, adding the computation of an unique reference frame to make it more robust.

OUR-CVFH relies on the use of Semi-Global Unique Reference Frames (SGURFs), which are repeatable coordinate systems computed for each region. Not only they remove the invariance to camera roll and allow to extract the 6DoF pose directly without additional steps, but they also improve the spatial descriptiveness.

The first part of the computation is akin to CVFH, but after segmentation, the points in each region are filtered once more according to the difference between their normals and the region's average normal. This results in better shaped regions, improving the estimation of the Reference Frames (RFs).

After this, the SGURF is computed for each region. Disambiguation is performed to decide the sign of the axes, according to the points' distribution. If this is not enough and the sign remains ambiguous, multiple RFs will need to be created to account for it. Finally, the OUR-CVFH descriptor is computed. The original Shape Distribution Component (SDC) is discarded, and the surface is now described according to the RFs.

[[Image:OUR-CVFH.png|thumb|center|600px| SGURF frame and resulting histogram of a region (image from [http://vision.deis.unibo.it/fede/papers/dagm12.pdf original paper]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/our_cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the OUR-CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// OUR-CVFH estimation object.
pcl::OURCVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> ourcvfh;
ourcvfh.setInputCloud(object);
ourcvfh.setInputNormals(normals);
ourcvfh.setSearchMethod(kdtree);
ourcvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
ourcvfh.setCurvatureThreshold(1.0);
ourcvfh.setNormalizeBins(false);
// Set the minimum axis ratio between the SGURF axes. At the disambiguation phase,
// this will decide if additional Reference Frames need to be created, if ambiguous.
ourcvfh.setAxisRatio(0.8);

ourcvfh.compute(*descriptors);
}</syntaxhighlight>

You can use the <span style="color:#FF1493">"getTransforms()"</span> function to get the transformations aligning the cloud to the corresponding SGURF.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points], [Axis ratio]
* '''Output''': OUR-CVFH descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/dagm12.pdf OUR-CVFH – Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram for Object Recognition and 6DOF Pose Estimation] (Aitor Aldoma et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_o_u_r_c_v_f_h_estimation.html pcl::OURCVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_OUR-CVFH.tar.gz Download]
</div>

==ESF==

The Ensemble of Shape Functions (ESF) is a combination of 3 different shape functions that describe certain properties of the cloud's points: distances, angles and area. This descriptor is very unique because it does not require normal information. Actually, it does not need any preprocessing, as it is robust to noise and incomplete surfaces.

The algorithm uses a voxel grid as an approximation of the real surface. It iterates through all the points in the cloud: for every iteration, 3 random points are chosen. For these points, the shape functions are computed:

* '''D2''': this function computes the distances between point pairs (3 overall). Then, for every pair, it checks if the line that connects both points lies entirely inside the surface, entirely outside (crossing free space), or both. Depending on this, the distance value will be binned to one of three possible histograms: IN, OUT or MIXED.
* '''D2 ratio''': an additional histogram for the ratio between parts of the line inside the surface, and parts outside. This value will be 0 if the line is completely outside, 1 if completely inside, and some value in between if mixed.
* '''D3''': this computes the square root of the area of the triangle formed by the 3 points. Like D2, the result is also classified as IN, OUT or MIXED, each with its own histogram.
* '''A3''': this function computes the angle formed by the points. Then, the value is binned depending on how the line opposite to the angle is (once again, as IN, OUT or MIXED).

[[Image:ESF.png|thumb|center|600px| ESF descriptor (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

After the loop is over, we are left with 10 subhistograms (IN, OUT and MIXED for D2, D3 and A3, and an additional one for the ratio). Each one has 64 bins, so the size of the final ESF descriptor is 640.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the ESF descriptor.
pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// ESF estimation object.
pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
esf.setInputCloud(object);

esf.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster)
* '''Output''': ESF descriptor
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6181760 Ensemble of shape functions for 3D object classification] (requires IEEE Xplore subscription) (Walter Wohlkinger and Markus Vincze, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_e_s_f_estimation.html pcl::ESFEstimation]
* [http://robotica.unileon.es/~victorm/PCL_ESF.tar.gz Download]
</div>

==GFPFH==

As you may have guessed, GFPFH stands for Global Fast Point Feature Histogram, the global version of the FPFH descriptor. GFPFH was designed for the task of helping a robot navigate its environment, having some context of the objects around it.

The first step before being able to compute the descriptor is surface categorization. A set of logical primitives (the classes, or categories) is created, which depends on the type of objects we expect the robot to find on the scene. For example, if we know there will be a coffee mug, we create three: one for the handle, and the other two for the outer and inner faces. Then, FPFH descriptors are computed, and everything is fed to a [https://en.wikipedia.org/wiki/Conditional_random_field Conditional Random Field] (CRF) algorithm. The CRF will label each surface with one of the previous categories, so we end up with a cloud where each point has been classified depending of the type of object (or object's region) it belongs to.

[[Image:FPFH_CRF.png|thumb|center|300px| Classification of objects made with FPFH and CRF (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

Now, the GFPFH descriptor can be computed with the result of the classification step. It will encode what the object is made of, so the robot can easily recognize it. First, an [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Octree | octree]] is created, dividing the object in voxel leaves. For every leaf, a set of probabilities is created, one for each class. Each one stores the probability of that leaf belonging to the class, and it is computed according to the number of points in that leaf that have been labelled as that class, and the total number of points. Then, for every pair of leaves in the octree, a line is casted, connecting them. Every leaf in its path is checked for occupancy, storing the result in an histogram. If the leaf is empty (free space), a value of 0 is saved. Otherwise, the leaf probabilities are used.

[[Image:GFPFH.png|thumb|center|500px| Computing the GFPFH with a voxel grid (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

The following code will compute the GFPFH for a cloud with label information. The categorization step is up to you, as it depends largely on the type of the scene, and the use you are going to give it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/gfpfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZL>::Ptr object(new pcl::PointCloud<pcl::PointXYZL>);
// Object for storing the GFPFH descriptor.
pcl::PointCloud<pcl::GFPFHSignature16>::Ptr descriptor(new pcl::PointCloud<pcl::GFPFHSignature16>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZL>(argv[1], *object) != 0)
{
return -1;
}

// Note: you should now perform classification on the cloud's points. See the
// original paper for more details. For this example, we will now consider 4
// different classes, and randomly label each point as one of them.
for (size_t i = 0; i < object->points.size(); ++i)
{
object->points[i].label = 1 + i % 4;
}

// ESF estimation object.
pcl::GFPFHEstimation<pcl::PointXYZL, pcl::PointXYZL, pcl::GFPFHSignature16> gfpfh;
gfpfh.setInputCloud(object);
// Set the object that contains the labels for each point. Thanks to the
// PointXYZL type, we can use the same object we store the cloud in.
gfpfh.setInputLabels(object);
// Set the size of the octree leaves to 1cm (cubic).
gfpfh.setOctreeLeafSize(0.01);
// Set the number of classes the cloud has been labelled with (default is 16).
gfpfh.setNumberOfClasses(4);

gfpfh.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Labels, Number of classes, Leaf size
* '''Output''': GFPFH descriptor
* '''Publication''':
** [https://www.willowgarage.com/sites/default/files/iccv09.pdf Detecting and Segmenting Objects for Mobile Manipulation] (Radu Bogdan Rusu et al., 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_f_p_f_h_estimation.html pcl::GFPFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GFPFH.tar.gz Download]
</div>

==GRSD==

The global version of the Radius-based Surface Descriptor works in a similar fashion to GFPFH. A voxelization and a surface categorization step are performed beforehand, labelling all surface patches according to the geometric category (plane, cylinder, edge, rim, sphere), using RSD. Then, the whole cluster is classified into one of these categories, and the GRSD descriptor is computed from this.

[[Image:GRSD.png|thumb|center|400px| Classification of objects for GRSD and resulting histogram (image from [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf original paper]).]]

To compute it:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/grsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the GRSD descriptors for each point.
pcl::PointCloud<pcl::GRSDSignature21>::Ptr descriptors(new pcl::PointCloud<pcl::GRSDSignature21>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// GRSD estimation object.
GRSDEstimation<PointXYZ, Normal, GRSDSignature21> grsd;
grsd.setInputCloud(cloud);
grsd.setInputNormals(normals);
grsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
grsd.setRadiusSearch(0.05);

grsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Radius
* '''Output''': GRSD descriptor
* '''Publications''':
** [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
** [http://www.mi.t.u-tokyo.ac.jp/top/downloadpublication/36 Voxelized Shape and Color Histograms for RGB-D] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_r_s_d_estimation.html pcl::GRSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GRSD.tar.gz Download]
</div>

=Saving and loading=

You can save a descriptor to a file just [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Writing_to_file|like with any other cloud type]]. One caveat, though. If you are using a descriptor that has its own custom type, like <span style="color:#FF1493">"PFHSignature125"</span>, everything will be OK. But with descriptors that do not (where you have to use <span style="color:#FF1493">"pcl::Histogram<>"</span>), you will get this error: <span style="color:#FF1493">''"POINT_TYPE_NOT_PROPERLY_REGISTERED"''</span>. In order to save or load from a file, PCL's IO functions need to know about the number, type and size of the fields. To solve this, you will have to properly register a new point type for your descriptor. For example, this will work for the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#RoPS|RoPS]] descriptor example we saw earlier:

<syntaxhighlight lang=CPP>POINT_CLOUD_REGISTER_POINT_STRUCT(ROPS135,
(float[135], histogram, histogram)
)</syntaxhighlight>

Add the previous to your code (change it accordingly), and you will be able to save and load descriptors as usual.

=Visualization=

Sometimes it is desired to check a visual representation of a descriptor, perhaps to analyze the distribution of data over different bins. Because they are saved as histograms, this is something trivial to do. PCL offers a couple of classes to do this.

==PCLHistogramVisualizer==

<span style="color:#FF1493">"PCLHistogramVisualizer"</span> is the simplest way to plot an histogram. The class has little functionality, but it does its job. Only one call is necessary to give the histogram and its size:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/histogram_visualizer.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLHistogramVisualizer viewer;
// We need to set the size of the descriptor beforehand.
viewer.addFeatureHistogram(*descriptor, 308);

viewer.spin();
}</syntaxhighlight>

[[Image:Histogram_visualizer_VFH.png|thumb|center|500px| VFH histogram seen with the Histogram Visualizer.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_histogram_visualizer.html pcl::visualization::PCLHistogramVisualizer]
* [http://robotica.unileon.es/~victorm/PCL_histogram_visualizer.tar.gz Download]
</div>

==PCLPlotter==

This class has all the methods from <span style="color:#FF1493">"PCLHistogramVisualizer"</span> (which will be deprecated soon) plus a lot more features. The code is almost the same:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/pcl_plotter.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLPlotter plotter;
// We need to set the size of the descriptor beforehand.
plotter.addFeatureHistogram(*descriptor, 308);

plotter.plot();
}</syntaxhighlight>

[[Image:PCL_plotter_VFH.png|thumb|center|500px| VFH histogram seen with the PCLPlotter class.]]

If you have raw data (such as a vector of floats) you can use the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html#aaf23b2b1c2f91c517cfde387ee1b654e addHistogramData()] function to plot it as an histogram.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pcl_plotter.php PCLPlotter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html pcl::visualization::PCLPlotter]
* [http://robotica.unileon.es/~victorm/PCL_plotter.tar.gz Download]
</div>

==PCL Viewer==

This program, included with PCL, will also let you open and visualize a saved descriptor. Internally, it uses [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#PCLPlotter|PCLPlotter]]. You can invoke the viewer from the command line like this, so it comes handy:

<syntaxhighlight lang=Bash>pcl_viewer <descriptor_file></syntaxhighlight>

[[Image:PCL_viewer_VFH.png|thumb|center|500px| VFH histogram seen with ''pcl_viewer''.]]

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 1: Installing and testing

2015-11-04T19:10:34Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Kinect.jpg|thumb|right|200px|Microsoft Kinect device.]]

This series of tutorials will explain the usage of a depth camera like [https://en.wikipedia.org/wiki/Kinect Kinect] for "serious" researching purposes. As you may know, Kinect is in fact an affordable depth sensor, developed with technology from [https://en.wikipedia.org/wiki/PrimeSense PrimeSense], based on infrarred structured light method. It also has a common camera (which makes it a RGB-D device), a microphone and a motorized pivot. Its use is not limited to playing with a Xbox360 console, you can plug it to a computer and use it like any other sensor. Many open-source drivers and frameworks are available.

Since its release on November 2010, it has gained a lot of popularity, specially among the scientific community. Many researches have procured themselves one because, despite the low cost (about 100 $), it has proven to be a powerful solution for depth sensing projects. Current investigations focus on real-time surface mapping, object recognition and tracking, and localization. Impressive results (like the [http://research.microsoft.com/en-us/projects/surfacerecon/ KinectFusion] project from Microsoft) are already possible.

The new [https://en.wikipedia.org/wiki/Xbox_One Xbox One] ships with an upgraded version, [https://en.wikipedia.org/wiki/Kinect_for_Xbox_One Kinect v2], with enhanced resolution, that is able to detect your facial expression, measure your heart rate, and track every one of your fingers. A PC development-ready version ([https://en.wikipedia.org/wiki/Kinect_for_Xbox_One#Kinect_for_Windows Kinect for Windows v2]) was released in July 2014, but it could only be used with the official Windows SDK (open source support [https://github.com/OpenKinect/libfreenect2 exists] but is still young). In October 2014 an adapter that allows to plug the standard Kinect v2 to a PC was released, so the development version of the sensor was discontinued in April 2015. Now you can just buy the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-for-Xbox-One/productID.307499400 standard one] for the console plus the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600 adapter].

I will explain the installation and usage of one of the "legacy" Kinect 1.0 devices with a common PC, and the possibilities it offers. I will do it in an easy to understand way, intended for students that have just acquired it and want to start from scratch.

Keep in mind that the software that we are going to use (Point Cloud Library and OpenNI drivers) will also let you use any other device like the [https://www.asus.com/3D-Sensor/Xtion_PRO/ Xtion PRO] or [https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/ Xtion PRO LIVE] from ASUS (the PRO only has a depth sensor, the PRO LIVE is a RGB-D camera) without changing a line of code.

<span style="color:#606060">'''''NOTE: The tutorials are written for Linux platforms. Also, 64-bit versions seem to work better than 32-bit.'''''</span>

=Requirements=

You will need the following:

* A common Kinect device, out of the box. You can buy it in your local electronics shop, or online. It also includes a free copy of ''[https://en.wikipedia.org/wiki/Kinect_Adventures! Kinect Adventures!]'', which is useless if you do not own the console. Microsoft has released a ''Kinect for Windows'' device, which is a normal looking Kinect no longer compatible with Xbox360, that will only work with their official SDK, intended for developers only. Also, like I stated earlier, you can use an ASUS Xtion indistinctly.
* A computer running Linux (Debian or Ubuntu preferably).
* A medium-sized room. Kinect has some limitations for depth measurement: 40cm minimum, 8m maximum (make it 6).

<span style="color:#606060">'''''NOTE: ''Kinect for Windows'' may have problems working with open source drivers on Linux .'''''</span>

=Connecting everything=

Kinect does not work with a common USB port. Its power consumption is a bit higher because of the motor, so Microsoft came up with a connector that combines USB and power supply. Old Xbox 360 models needed a special adapter, new S model ones already have this new port. Luckily, Kinect comes with the official adapter out of the box (otherwise you will have to [http://www.amazon.com/Power-Supply-Cable-Kinect-Xbox-360/dp/B004S7GA46/ref=sr_1_1?ie=UTF8&qid=1378288327&sr=8-1 buy one]).

Just plug the adapter to any power socket, and the USB to your computer. Let's check typing this in a terminal:

<syntaxhighlight lang=Bash>lsusb</syntaxhighlight>

Output should list the following devices:

<syntaxhighlight lang=Bash>Bus 001 Device 005: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 006: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 007: ID 045e:02ae Microsoft Corp. Xbox NUI Camera</syntaxhighlight>

If you are using a Xtion, you should see:

<syntaxhighlight lang=Bash>Bus 001 Device 004: ID 1d27:0601 ASUS</syntaxhighlight>

"0601" identifies the new Xtion model, "0600" the older one. Both should work the same. But try to avoid [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO|USB 3.0 ports]]!

=Installing the software=

There is more than one way to get your Kinect working on your PC, and start developing applications for it:

* [https://www.microsoft.com/en-us/download/details.aspx?id=40278 Kinect for Windows SDK] and [https://www.microsoft.com/en-us/download/details.aspx?id=40276 Developer Toolkit]: released on June 16, 2011 as a non-commercial SDK intended for application development. Version 1.8, the last there will ever be now that Kinect v2 is out, was released on September 2013. Because it comes from Microsoft, it is obviously the easiest way to get everything working. Sadly, there is no Linux version.
* [https://github.com/OpenKinect/libfreenect libfreenect] library: from the [http://openkinect.org/wiki/Main_Page OpenKinect] project, it is intended to be a free and open source alternative to the official drivers. libfreenect is used by projects like [https://github.com/ofTheo/ofxKinect ofxKinect], an addon for the [http://www.openframeworks.cc/ openFrameworks] toolkit (and as of version 0.8, included in the core package) that runs on Linux and OS X. ofxKinect packs a nice example application to show the RGB and point cloud taken from Kinect.
* [https://github.com/PrimeSense/Sensor PrimeSense drivers]: they were released as open source after the the [https://en.wikipedia.org/wiki/OpenNI OpenNI] project was created, along with the motion tracking middleware, ''NITE'', and the SDK. NI stands for Natural Interaction, and the project tried to enforce a common standard for human input using Kinect-like sensors. These official drivers are used by [http://wiki.ros.org/openni_kinect ROS] (the Robot Operating System, a massive collection of libraries and tools for robotic researchers) and [http://pointclouds.org/ PCL] (the Point Cloud Library, with everything needed for 3D point cloud processing). Sadly, version 2.0 of the OpenNI SDK dropped support for Kinect on Linux due to licensing issues, so for now PCL relies on legacy 1.x versions. Also, Apple bought PrimeSense on November 2013, and on April 2014 OpenNI's webpage was closed. The source is now being maintained by [http://structure.io/openni a third party].
* [https://github.com/avin2/SensorKinect SensorKinect]: a modified version of the official PrimeSense drivers, used for example by [https://github.com/gameoverhack/ofxOpenNI ofxOpenNI] (another openFrameworks addon). Last updated on 2012.

For this tutorial, we are going to use PCL with the OpenNI drivers, so owners of a Xtion can also get it to work.

==Precompiled PCL for Ubuntu==

If you have a valid [http://wiki.ros.org/ROS/Installation installation of ROS] (through their repository), you do not have to install anything. Both OpenNI and PrimeSense drivers, as well as PCL, should be already installed. You can check it with:

<syntaxhighlight lang=Bash>sudo apt-get install libpcl-1.7-all libpcl-1.7-all-dev libopenni-dev libopenni-sensor-primesense-dev</syntaxhighlight>

The previous command should state that all packages are already installed (change the PCL version number as needed), install them if not. If you get an error about overwriting some file, check [[PCL/OpenNI_troubleshooting#Trying_to_overwrite_X.2C_which_is_also_in_package_Y|this]].

In case you do not want ROS, there is a [https://launchpad.net/~v-launchpad-jochen-sprickerhof-de/+archive/ubuntu/pcl PPA] (Personal Package Archive, a private repository) which has everything we need. Add it to your sources, and install everything:

<syntaxhighlight lang=Bash>sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install build-essential cmake libpcl1.7 libpcl-dev pcl-tools</syntaxhighlight>

Trying to mix ROS and PCL repositories and packages can cause some errors, so choose one of them and stick with it. Check the [[PCL/OpenNI troubleshooting]] page because your Kinect may not work by default in 32 bits.

==Compiling PCL from source==

For Linuxes without a precompiled version of PCL, you will need to compile it yourself. This has an advantage, actually: you can customize the build options and choose what you want. Also, the resulting binaries and libraries should be a bit faster. And you always get the latest version! The instructions are [http://pointclouds.org/documentation/tutorials/compiling_pcl_posix.php here], but the steps are easy so I will show them to you.

===Installing the dependencies===

Some of PCL dependencies can be installed via the package manager. Others will require additional work.

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install build-essential cmake cmake-curses-gui libboost-all-dev libeigen3-dev libflann-dev libvtk5-dev libvtk5-qt4-dev libglew-dev libxmu-dev libsuitesparse-dev libqhull-dev libpcap-dev libxmu-dev libxi-dev libgtest-dev libqt4-opengl-dev</syntaxhighlight>

The trunk (1.8) version of PCL uses [https://en.wikipedia.org/wiki/VTK VTK] 6 and [https://en.wikipedia.org/wiki/Qt_%28software%29 Qt] 5, so if you intend to compile it, you must install the following packages (say yes if you are asked to remove VTK 5 and Qt 4 packages):

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

====JDK====

OpenNI requires Sun's official JDK (Java Development Kit), which is no longer available on official apt repositories. You can use [http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html unofficial ones], or do it manually. For the last method, go to the [http://www.oracle.com/technetwork/java/javase/downloads/index.html Java SE downloads] page (SE means Standard Edition) and download the latest version (e.g., <span style="color:#1E90FF">''jdk-8u66-linux-x64.tar.gz''</span>). Extract it, then move the contents to <span style="color:#FF8C00">''/usr/lib/jvm/''</span> so it is available system-wide:

<syntaxhighlight lang=Bash>sudo mkdir -p /usr/lib/jvm/
sudo cp -r jdk1.8.0_66/ /usr/lib/jvm/</syntaxhighlight>

Then, make it the default choice to compile and run Java programs. Remember to change the version number as needed!

<syntaxhighlight lang=Bash>sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_66/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_66/bin/javac" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.8.0_66/bin/jar" 1</syntaxhighlight>

To be sure, use the following commands to manually select the correct option, in case there is more than one choice:

<syntaxhighlight lang=Bash>sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config jar</syntaxhighlight>

If you are still not sure, run them and display the version, making sure it is the one you installed:

<syntaxhighlight lang=Bash>java -version
javac -version</syntaxhighlight>

Sun's JDK is now installed.

====OpenNI====

PCL uses OpenNI and the PrimeSense drivers to get data from devices like Kinect or Xtion. It is optional, but it would not make much sense not to install it, would it? If you are using Ubuntu, add the [[#Precompiled PCL for Ubuntu | PPA]] and install <span style="color:#228B22">''libopenni-dev''</span> and <span style="color:#228B22">''libopenni-sensor-primesense-dev''</span>, which should be done already. Otherwise, fetch the [https://github.com/OpenNI/OpenNI OpenNI] and [https://github.com/PrimeSense/Sensor PrimeSense Sensor] sources from GitHub (download them as .zip, the link is on the right). Extract them, and install the dependencies:

<syntaxhighlight lang=Bash>sudo apt-get install python libusb-1.0-0-dev freeglut3-dev doxygen graphviz</syntaxhighlight>

Go to the directory where you extracted OpenNI (<span style="color:#FF8C00">''OpenNI-master/''</span> for me), and open a terminal in the <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span> subdirectory. Issue:

<syntaxhighlight lang=Bash>./RedistMaker</syntaxhighlight>

When it finishes, and if there are no errors (check the [[PCL/OpenNI troubleshooting]] page if you get any), go to <span style="color:#FF8C00">''Platform/Linux/Redist/OpenNI-Bin-Dev-Linux-x64-v1.5.7.10/''</span> (or your equivalent), and install (you must be root):

<syntaxhighlight lang=Bash>sudo ./install.sh</syntaxhighlight>

Now, go to the directory where you extracted the PrimeSense drivers (<span style="color:#FF8C00">''Sensor-master/''</span> for me), and repeat the exact same procedure (go to <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span>, issue <span style="color:#228B22">''./RedistMaker''</span>, go to <span style="color:#FF8C00">''Platform/Linux/Redist/Sensor-Bin-Linux-x64-v5.1.6.6/''</span>, issue <span style="color:#228B22">''sudo ./install.sh''</span>). Congratulations, you have now installed OpenNI.

====CUDA====

Like OpenNI, [https://en.wikipedia.org/wiki/CUDA nVidia CUDA] is an optional dependency, that will allow PCL to use your GPU (Graphics Processing Unit, that is, your graphics card) for certain computations. This is mandatory for tools like KinFu (do not bother unless you have at least a series 400 card with 1.5 GB of VRAM).

Some distributions provide packages for CUDA in the repositories. For example, in Ubuntu:

<syntaxhighlight lang=Bash>sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit</syntaxhighlight>

If you want to install it manually (which is incompatible with the previous method), go to the [https://developer.nvidia.com/cuda-downloads CUDA downloads] page, which is self-explanatory, and get the small .deb file or the huge .run toolkit/SDK installer file for your system (you should have installed the nVidia drivers already, but the installer will also do it for you if needed).

If you chose the .deb file, install it using the method of your choice, like Gdebi package manager, or through console:

<syntaxhighlight lang=Bash>sudo dpkg -i <package.deb></syntaxhighlight>

The .deb does not contain all CUDA stuff, it just adds their repository to your software lists. Now you must install everything:

<syntaxhighlight lang=Bash>sudo apt-get update
sudo apt-get install cuda</syntaxhighlight>

If, on the other hand, you downloaded the .run, give it execution permissions:

<syntaxhighlight lang=Bash>chmod +x cuda_7.0.28_linux.run</syntaxhighlight>

And install it. You can use the default options, although if you have a working nVidia graphics driver for your card, you may want to say "no" when the installer offers to install it for you:

<syntaxhighlight lang=Bash>sudo ./cuda_7.0.28_linux.run</syntaxhighlight>

The environment variables need to be changed so your system can find CUDA's libraries and binaries. Open <span style="color:#1E90FF">''/etc/ld.so.conf''</span>:

<syntaxhighlight lang=Bash>sudo nano /etc/ld.so.conf</syntaxhighlight>

And append one of these two lines:

<syntaxhighlight lang=Bash>/usr/local/cuda/lib64 # Add this on 64-bit only.
/usr/local/cuda/lib # Add this on 32-bit only.</syntaxhighlight>

Save with <span style="color:#228B22">'''Ctrl+O'''</span> and Enter, exit with <span style="color:#228B22">'''Ctrl+X'''</span>. Reload the cache of the dynamic linker with:

<syntaxhighlight lang=Bash>sudo ldconfig</syntaxhighlight>

Now, append CUDA's bin directory to your PATH. Do this by editing your local <span style="color:#1E90FF">''.bashrc''</span> file:

<syntaxhighlight lang=Bash>nano ~/.bashrc</syntaxhighlight>

And append this line:

<syntaxhighlight lang=Bash>export PATH=$PATH:/usr/local/cuda/bin</syntaxhighlight>

CUDA is now installed.

===Getting the source===

Every dependency is installed. Time to download PCL's source code. First, you must choose whether to install the stable or the experimental branch of PCL. The stable branch is the latest official release and it is guaranteed to work without problems. The experimental branch may give you a compiling error seldomly, but you can find some interesting features that stable users will have to wait some months for. Apart from that, both are built the same way.

To get the stable version, go to the [https://github.com/PointCloudLibrary/pcl/releases downloads] page, get <span style="color:#1E90FF">''pcl-pcl-1.7.2.tar.gz''</span> or whatever the latest release is, and extract it somewhere. For the experimental version, use Git:

<syntaxhighlight lang=Bash>sudo apt-get install git
git clone https://github.com/PointCloudLibrary/pcl PCL-trunk-Source</syntaxhighlight>

The compiled trunk version of PCL will take up more than 8 GB. So make sure you put the source folder in a partition with enough free space!

===Compiling===

Go the the PCL source directory (<span style="color:#FF8C00">''pcl-pcl-1.7.2/''</span> or <span style="color:#FF8C00">''PCL-trunk-Source/''</span>), and create a new subdirectory to keep the build files in:

<syntaxhighlight lang=Bash>mkdir build
cd build</syntaxhighlight>

Now it is time to configure the project using CMake. We will tell it to build in Release (fully optimized, no debug capabilities) mode now, and customize the rest of the options later:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..</syntaxhighlight>

CMake should be able to find every dependency, thus being able to build every subsystem except for the ones marked as <span style="color:#FF1493">''"Disabled by default"''</span>. If you are happy, you can build now, otherwise let's invoke CMake's curses interface to change a couple of things (mind the final dot):

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

[[Image:Ccmake_GUI_PCL.png|thumb|center|900px|Interface of ''ccmake''.]]

Here you can change the build options. The program usage can be found at the bottom of the screen. Try turning all functionality <span style="color:#FF1493">"ON"</span>. The most important thing, in case you want to use CUDA, is to enable it and give CMake the path to your SDK. Go to the <span style="color:#FF1493">"CUDA_SDK_ROOT_DIR"</span> option and enter the correct path (probably <span style="color:#FF8C00">''/usr/local/cuda/''</span> or similar).

When you are done, press <span style="color:#228B22">'''C'''</span> to configure and <span style="color:#228B22">'''G'''</span> to generate and exit the tool. Sometimes, the options you change can activate previously omitted parameters, or prompt some warning text. Just press <span style="color:#228B22">'''E'''</span> when you are finished reading the message, and keep pressing <span style="color:#228B22">'''C'''</span> until it lets you generate (new parameters will be marked with an asterisk, so you can check them and decide whether or not you want further customization).

If you are done configuring, it is time to build:

<syntaxhighlight lang=Bash>make</syntaxhighlight>

<span style="color:#606060">'''''NOTE: Additionally, you can append the parameter -jX to speed up the compilation, X being the number of cores or processors of your PC, plus one.'''''</span>

Remember that, at any time, you can manually force the project to be reconfigured and built from scratch by emptying the <span style="color:#FF8C00">''build/''</span> directory with:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

===Installing===

It will take some time to compile PCL (up to a few hours if your PC is not powerful enough). When it is finished, install it system-wide with:

<syntaxhighlight lang=Bash>sudo make install</syntaxhighlight>

And you should reboot and proceed to the next section, to see if your computer now recognizes (and uses) your Kinect device.

=Testing (OpenNI viewer)=

We are going to write a simple example program that will fetch data from the Kinect or Xtion and present it to the user, using the PCL library. It will also allow to save the current frame (as point cloud) to disk. If you feel lazy, you can download it [http://robotica.unileon.es/~victorm/PCL_openni_viewer.tar.gz here], and skip the next two sections. Otherwise, create a new directory anywhere in your hard disk and proceed.

==CMakeLists.txt==

Inside that directory, create a new text file named <span style="color:#1E90FF">''CMakeLists.txt''</span>. PCL-based programs use the CMake build system, too. Open it with any editor and paste the following content:

<syntaxhighlight lang=CMake>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

project(PCL_openni_viewer)

find_package(PCL 1.7 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

set(PCL_BUILD_TYPE Release)

file(GLOB PCL_openni_viewer_SRC
"src/*.h"
"src/*.cpp"
)
add_executable(openniViewer ${PCL_openni_viewer_SRC})

target_link_libraries (openniViewer ${PCL_LIBRARIES})</syntaxhighlight>

CMake syntax is quite self-explanatory. We ask for a CMake version 2.8 installation, minimum. We declare a new project named <span style="color:#FF1493">"openni_PCL_viewer"</span>. We tell CMake to check for the presence of PCL library development files, version 1.7. If our system can not meet the CMake and PCL version requirement, the process will fail.

Next, we feed the compiler and linker the directories where PCL includes and libraries can be found, and the defined symbols. We tell CMake to use the <span style="color:#FF1493">"Release"</span> build type, which will activate certain optimizations depending on the compiler we use. Other build types are available, like <span style="color:#FF1493">"Debug"</span>, <span style="color:#FF1493">"MinSizeRel"</span>, and <span style="color:#FF1493">"RelWithDebInfo"</span>.

Finally, we create a variable, <span style="color:#FF1493">"opennipclviewer_SRC"</span>, that will store a list of files to be compiled (though we will only have one). We create a new binary to be compiled from these source files, and we link it with the PCL library.

Check the [https://cmake.org/Wiki/CMake_Useful_Variables CMake help] for more interesting options.

==main.cpp==

We told CMake it could find the source files in a <span style="color:#FF8C00">''src/''</span> subdirectory, so let's keep to out word and create it. Then, add a new <span style="color:#1E90FF">''main.cpp''</span> file inside and paste the following lines:

<syntaxhighlight lang=CPP>// Original code by Geoffrey Biggs, taken from the PCL tutorial in
// http://pointclouds.org/documentation/tutorials/pcl_visualizer.php

// Simple OpenNI viewer that also allows to write the current scene to a .pcd
// when pressing SPACE.

#include <pcl/io/openni_grabber.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/console/parse.h>

#include <iostream>

using namespace std;
using namespace pcl;

PointCloud<PointXYZRGBA>::Ptr cloudptr(new PointCloud<PointXYZRGBA>); // A cloud that will store color info.
PointCloud<PointXYZ>::Ptr fallbackCloud(new PointCloud<PointXYZ>); // A fallback cloud with just depth data.
boost::shared_ptr<visualization::CloudViewer> viewer; // Point cloud viewer object.
Grabber* openniGrabber; // OpenNI grabber that takes data from the device.
unsigned int filesSaved = 0; // For the numbering of the clouds saved to disk.
bool saveCloud(false), noColor(false); // Program control.

void
printUsage(const char* programName)
{
cout << "Usage: " << programName << " [options]"
<< endl
<< endl
<< "Options:\n"
<< endl
<< "\t<none> start capturing from an OpenNI device.\n"
<< "\t-v FILE visualize the given .pcd file.\n"
<< "\t-h shows this help.\n";
}

// This function is called every time the device has new data.
void
grabberCallback(const PointCloud<PointXYZRGBA>::ConstPtr& cloud)
{
if (! viewer->wasStopped())
viewer->showCloud(cloud);

if (saveCloud)
{
stringstream stream;
stream << "inputCloud" << filesSaved << ".pcd";
string filename = stream.str();
if (io::savePCDFile(filename, *cloud, true) == 0)
{
filesSaved++;
cout << "Saved " << filename << "." << endl;
}
else PCL_ERROR("Problem saving %s.\n", filename.c_str());

saveCloud = false;
}
}

// For detecting when SPACE is pressed.
void
keyboardEventOccurred(const visualization::KeyboardEvent& event,
void* nothing)
{
if (event.getKeySym() == "space" && event.keyDown())
saveCloud = true;
}

// Creates, initializes and returns a new viewer.
boost::shared_ptr<visualization::CloudViewer>
createViewer()
{
boost::shared_ptr<visualization::CloudViewer> v
(new visualization::CloudViewer("OpenNI viewer"));
v->registerKeyboardCallback(keyboardEventOccurred);

return (v);
}

int
main(int argc, char** argv)
{
if (console::find_argument(argc, argv, "-h") >= 0)
{
printUsage(argv[0]);
return -1;
}

bool justVisualize(false);
string filename;
if (console::find_argument(argc, argv, "-v") >= 0)
{
if (argc != 3)
{
printUsage(argv[0]);
return -1;
}

filename = argv[2];
justVisualize = true;
}
else if (argc != 1)
{
printUsage(argv[0]);
return -1;
}

// First mode, open and show a cloud from disk.
if (justVisualize)
{
// Try with color information...
try
{
io::loadPCDFile<PointXYZRGBA>(filename.c_str(), *cloudptr);
}
catch (PCLException e1)
{
try
{
// ...and if it fails, fall back to just depth.
io::loadPCDFile<PointXYZ>(filename.c_str(), *fallbackCloud);
}
catch (PCLException e2)
{
return -1;
}

noColor = true;
}

cout << "Loaded " << filename << "." << endl;
if (noColor)
cout << "This cloud has no RGBA color information present." << endl;
else cout << "This cloud has RGBA color information present." << endl;
}
// Second mode, start fetching and displaying frames from the device.
else
{
openniGrabber = new OpenNIGrabber();
if (openniGrabber == 0)
return -1;
boost::function<void (const PointCloud<PointXYZRGBA>::ConstPtr&)> f =
boost::bind(&grabberCallback, _1);
openniGrabber->registerCallback(f);
}

viewer = createViewer();

if (justVisualize)
{
if (noColor)
viewer->showCloud(fallbackCloud);
else viewer->showCloud(cloudptr);
}
else openniGrabber->start();

// Main loop.
while (! viewer->wasStopped())
boost::this_thread::sleep(boost::posix_time::seconds(1));

if (! justVisualize)
openniGrabber->stop();
}</syntaxhighlight>

Save and close.

==Compiling==

Follow the same steps you used to build PCL. That is, create a new <span style="color:#FF8C00">''build/''</span> subdirectory next to the <span style="color:#FF8C00">''src/''</span> one. Open a terminal there and issue:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..
make</syntaxhighlight>

==Executing==

Still from the same terminal, run the compiled example program:

<syntaxhighlight lang=Bash>./openniViewer</syntaxhighlight>

After some seconds, the main window will appear and the application will start grabbing frames from the device. You can inspect the current point cloud using the mouse, holding the left button to rotate, the right one (or the mouse wheel) to zoom, and the middle one to pan the camera around. At first, you may see only a black screen, or some big colored axes, but no cloud. Try zooming out, to see the whole scene. Another useful key is <span style="color:#228B22">'''R'''</span>, which will reset the camera parameters when pressed. Use it whenever you notice that zooming has gotten slow after some camera movement, or if you still can not see the cloud. See the [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer] tutorial for additional controls and features.

Whenever you feel ready, press the <span style="color:#228B22">'''SPACE'''</span> key. The program will pause for a fraction of a second and the output <span style="color:#FF1493">''"Saved inputCloud0.pcd."''</span> will appear on the console. Check the current folder to see that file <span style="color:#1E90FF">''inputCloud0.pcd''</span> has indeed been written. You can now close the program with <span style="color:#228B22">'''Q'''</span> or <span style="color:#228B22">'''Alt+F4'''</span>.

Next, run it again giving the following parameter:

<syntaxhighlight lang=Bash>./openniViewer -v inputCloud0.pcd</syntaxhighlight>

This will tell the program not to take data from the device, but from the saved point cloud file instead. After it loads, you will realize that you are presented the same scene you saved to disk.

<span style="color:#606060">'''''NOTE: PCD data is saved relative to the sensor. No matter how much you have manipulated the view, it will reset to default when you load the file.'''''</span>

=Conclusion=

At this point, your Kinect device should be working and getting depth data for you. There is a collection of excellent [http://pointclouds.org/documentation/tutorials/ tutorials] for PCL in the official webpage. I encourage you to finish them all before proceeding your experiments with the Kinect sensor. You can also find a good introduction/tutorial at the PCL library [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Introduction.pdf here].

If you use an ASUS Xtion PRO device instead, you should have gotten everything to work without problems or additional steps (except maybe for [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO | this one]]).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 4: 3D object recognition (descriptors)

2015-11-02T11:12:00Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

It is time to learn the basics of one of the most interesting applications of point cloud processing: 3D object recognition. Akin to 2D recognition, this technique relies on finding good keypoints (characteristic points) in the cloud, and matching them to a set of previously saved ones. But 3D has several advantages over 2D: namely, we will be able to estimate with decent accuracy the exact position and orientation of the object, relative to the sensor. Also, 3D object recognition tends to be more robust to clutter (crowded scenes where objects in the front occluding objects in the background). And finally, having information about the object's shape will help with collision avoidance or grasping operations.

In this first tutorial we will see what descriptors are, how many types are there available in PCL, and how to compute them.

=Overview=

The basis of 3D object recognition is to find a set of correspondences between two different clouds, one of them containing the object we are looking for. In order to do this, we need a way to compare points in an unambiguous manner. Until now, we have worked with points that store the XYZ coordinates, the RGB color... but none of those properties are unique enough. In two sequential scans, two points could share the same coordinates despite belonging to different surfaces, and using the color information takes us back to 2D recognition, will all the lightning related problems.

In a previous tutorial, we talked about [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Feature_estimation | features]], before introducing the normals. Normals are an example of feature, because they encode information about the vicinity of the point. That is, the neighboring points are taken into account when computing it, giving us an idea of how the surrounding surface is. But this is not enough. For a feature to be optimal, it must meet the following criteria:

# '''It must be robust to transformations''': rigid transformations (the ones that do not change the distance between points) like translations and rotations must not affect the feature. Even if we play with the cloud a bit beforehand, there should be no difference.
# '''It must be robust to noise''': measurement errors that cause noise should not change the feature estimation much.
# '''It must be resolution invariant''': if sampled with different density (like after performing downsampling), the result must be identical or similar.

This is where descriptors come in. They are more complex (and precise) signatures of a point, that encode a lot of information about the surrounding geometry. The purpose is to unequivocally identify a point across multiple point clouds, no matter the noise, resolution or transformations. Also, some of them capture additional data about the object they belong to, like the viewpoint (that lets us retrieve the pose).

[[Image:Correspondence.jpg|thumb|center|600px|Finding correspondences between point features of two clouds (image from http://pointclouds.org/).]]

There are many 3D descriptors implemented into PCL. Each one has its own method for computing unique values for a point. Some use the difference between the angles of the normals of the point and its neighbors, for example. Others use the distances between the points. Because of this, some are inherently better or worse for certain purposes. A given descriptor may be scale invariant, and another one may be better with occlusions and partial views of objects. Which one you choose depends on what you want to do.

After calculating the necessary values, an additional step is performed to reduce the descriptor size: the result is binned into an [https://en.wikipedia.org/wiki/Histogram histogram]. To do this, the value range of each variable that makes up the descriptor is divided into ''n'' subdivisions, and the number of occurrences in each one is counted. Try to imagine a descriptor that computes a single variable, that ranges from 1 to 100. We choose to create 10 bins for it, so the first bin would gather all occurrences between 1 and 10, the second from 11 to 20, and so on. We look at the value of the variable for the first point-neighbor pair, and it is 27, so we increment the value of the third bin by 1. We keep doing this until we get a final histogram for that keypoint. The bin size must be carefully chosen depending on how descriptive that variable is (the variables do not have to share the same number of bins, and also the bins do not have to be of the same size; if for example most values from the previous example fell in the 50-100 range then it would be sensible to have more bins of smaller size in that range).

Descriptors can be classified in two main categories: global and local. The process for computing and using each one (recognition pipeline) is different, so each will be explained in its own section in this article.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/how_features_work.php How 3D Features work in PCL]
** [https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features Overview and Comparison of Features]
** [http://www.pointclouds.org/assets/icra2013/pcl_features_icra13.pdf How does a good feature look like?]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

==Table==

The following table will give you a hint of how many descriptors are there in PCL, and some of their features:

{| class="wikitable sortable" style="margin: auto; text-align:center" cellpadding="15"
|+ 3D descriptors
|- style="vertical-align:middle;"
! scope="col"| Name
! scope="col"| Type
! scope="col"| Size<sup>†</sup>
! scope="col"| Custom PointType<sup>††</sup>
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#PFH|PFH]] (Point Feature Histogram)
| style="color: sienna;" | Local
| 125
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#FPFH|FPFH]] (Fast Point Feature Histogram)
| style="color: sienna;" | Local
| 33
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RSD|RSD]] (Radius-Based Surface Descriptor)
| style="color: sienna;" | Local
| 289
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#3DSC|3DSC]] (3D Shape Context)
| style="color: sienna;" | Local
| 1980
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#USC|USC]] (Unique Shape Context)
| style="color: sienna;" | Local
| 1960
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#SHOT|SHOT]] (Signatures of Histograms of Orientations)
| style="color: sienna;" | Local
| 352
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Spin_image|Spin image]]
| style="color: sienna;" | Local
| 153*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RIFT|RIFT]] (Rotation-Invariant Feature Transform)
| style="color: sienna;" | Local
| 32*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#NARF|NARF]] (Normal Aligned Radial Feature)
| style="color: sienna;" | Local
| 36
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RoPS|RoPS]] (Rotational Projection Statistics)
| style="color: sienna;" | Local
| 135*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#VFH|VFH]] (Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#CVFH|CVFH]] (Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#OUR-CVFH|OUR-CVFH]] (Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#ESF|ESF]] (Ensemble of Shape Functions)
| style="color: darkcyan;" | Global
| 640
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GFPFH|GFPFH]] (Global Fast Point Feature Histogram)
| style="color: darkcyan;" | Global
| 16
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GRSD|GRSD]] (Global Radius-Based Surface Descriptor)
| style="color: darkcyan;" | Global
| 21
| style="color: green;" | Yes
|}

† Values marked with an asterisk (*) indicate that the descriptor's size depends on some parameter(s), and the one given is for the default values.

†† Descriptors without their own custom PointType use the generic <span style="color:#FF1493">"pcl::Histogram<>"</span> type. See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|Saving and loading]].

Optionally, you can download a document with a less simple version of the table. Page format is A4, landscape:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Downloads''':
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.odt Table as original ODT] (uses font embedding, requires LibreOffice 4)
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.pdf Table as PDF]
</div>

=Local descriptors=

Local descriptors are computed for individual points that we give as input. They have no notion of what an object is, they just describe how the local geometry is around that point. Usually, it is your task to choose which points you want a descriptor to be computed for: the ''keypoints''. Most of the time, you can get away by just performing a downsampling and choosing all remaining points, but keypoint detectors are available, like the one used for [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF]], or [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#ISS|ISS]].

Local descriptors are used for object recognition and [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration | registration]]. Now we will see which ones are implemented into PCL.

==PFH==

PFH stands for Point Feature Histogram. It is one of the most important descriptors offered by PCL and the basis of others such as FPFH. The PFH tries to capture information of the geometry surrounding the point by analyzing the difference between the directions of the normals in the vicinity (and because of this, an imprecise normal estimation may produce low-quality descriptors).

First, the algorithm pairs all points in the vicinity (not just the chosen keypoint with its neighbors, but also the neighbors with themselves). Then, for each pair, a [https://en.wikipedia.org/wiki/Darboux_frame fixed coordinate frame] is computed from their normals. With this frame, the difference between the normals can be encoded with 3 angular variables. These variables, together with the euclidean distance between the points, are saved, and then binned to an histogram when all pairs have been computed. The final descriptor is the concatenation of the histograms of each variable (4 in total).

<center><gallery widths=300px>
File:PFH_neighbors.png | Point pairs established when computing the PFH for a point (image from http://pointclouds.org).
File:PFH_frame.png | Fixed coordinate frame and angular features computed for one of the pairs (image from http://pointclouds.org).
</gallery></center>

Computing descriptors in PCL is very easy, and the PFH is not an exception:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/pfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the PFH descriptors for each point.
pcl::PointCloud<pcl::PFHSignature125>::Ptr descriptors(new pcl::PointCloud<pcl::PFHSignature125>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// PFH estimation object.
pcl::PFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::PFHSignature125> pfh;
pfh.setInputCloud(cloud);
pfh.setInputNormals(normals);
pfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
pfh.setRadiusSearch(0.05);

pfh.compute(*descriptors);
}</syntaxhighlight>

As you can see, PCL uses the <span style="color:#FF1493">"PFHSignature125"</span> type to save the descriptor to. This means that the descriptor's size is 125 (the dimensionality of the feature vector). Dividing a feature in ''D'' dimensional space in ''B'' divisions requires a total of B<sup>D</sup> bins. The original proposal makes use of the distance between the points, but the implementation of PCL does not, as it was not considered discriminative enough (specially in 2.5D scans, where the distance between points increases the further away from the sensor). Hence, the remaining features (with one dimension each) can be divided in 5 divisions, resulting in a 125-dimensional (5<sup>3</sup>) vector.

The final object that stores the computed descriptors can be handled like a normal cloud (even saved to, or read from, disk), and it has the same number of "points" than the original one. The <span style="color:#FF1493">"PFHSignature125"</span> object at position 0, for example, stores the PFH descriptor for the <span style="color:#FF1493">"PointXYZ"</span> point at the same position in the cloud.

For additional details about the descriptor, check the original publications listed below, or PCL's tutorial.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': PFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pfh_estimation.php Point Feature Histograms (PFH) descriptors]
* '''Publications''':
** [http://ias.in.tum.de/_media/spezial/bib/rusu08ias.pdf Persistent Point Feature Histograms for 3D Point Clouds] (Radu Bogdan Rusu et al., 2008)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.4, page 50)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_p_f_h_estimation.html pcl::PFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_PFH.tar.gz Download]
</div>

===FPFH===

PFH gives accurate results, but it has a drawback: it is too computationally expensive to perform at real time. For a cloud of ''n'' keypoints with ''k'' neighbors considered, it has a [https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations complexity] of ''O(nk<sup>2</sup>)''. Because of this, a derived descriptor was created, named FPFH (Fast Point Feature Histogram).

The FPFH considers only the direct connections between the current keypoint and its neighbors, removing additional links between neighbors. This takes the complexity down to ''O(nk)''. Because of this, the resulting histogram is referred to as SPFH (Simplified Point Feature Histogram). The reference frame and the angular variables are computed as always.

[[Image:FPFH_neighbors.png|thumb|center|400px|Point pairs established when computing the FPFH for a point (image from http://pointclouds.org).]]

To account for the loss of these extra connections, an additional step takes place after all histograms have been computed: the SPFHs of a point's neighbors are "merged" with its own, weighted according to the distance. This has the effect of giving a point surface information of points as far away as 2 times the radius used. Finally, the 3 histograms (distance is not used) are concatenated to compose the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/fpfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the FPFH descriptors for each point.
pcl::PointCloud<pcl::FPFHSignature33>::Ptr descriptors(new pcl::PointCloud<pcl::FPFHSignature33>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// FPFH estimation object.
pcl::FPFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::FPFHSignature33> fpfh;
fpfh.setInputCloud(cloud);
fpfh.setInputNormals(normals);
fpfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
fpfh.setRadiusSearch(0.05);

fpfh.compute(*descriptors);
}</syntaxhighlight>

An additional implementation of the FPFH estimation that takes advantage of multithreaded optimizations (with the [https://en.wikipedia.org/wiki/OpenMP OpenMP API]) is available in the class <span style="color:#FF1493">"FPFHEstimationOMP"</span>. Its interface is identical to the standard unoptimized implementation. Using it will result in a big performance boost on multi-core systems, meaning faster computation times. Remember to include the header <span style="color:#FF1493">"fpfh_omp.h"</span> instead.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': FPFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/fpfh_estimation.php Fast Point Feature Histograms (FPFH) descriptors]
* '''Publications''':
** [http://files.rbrusu.com/publications/Rusu09ICRA.pdf Fast Point Feature Histograms (FPFH) for 3D Registration] (Radu Bogdan Rusu et al., 2009)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.5, page 57)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation.html pcl::FPFHEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation_o_m_p.html pcl::FPFHEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_FPFH.tar.gz Download]
</div>

==RSD==

The Radius-Based Surface Descriptor encodes the radial relationship of the point and its neighborhood. For every pair of the keypoint with a neighbor, the algorithm computes the distance between them, and the difference between their normals. Then, by assuming that both points lie on the surface of a sphere, said sphere is found by fitting not only the points, but also the normals (otherwise, there would be infinite possible spheres). Finally, from all the point-neighbor spheres, only the ones with the maximum and minimum radii are kept and saved to the descriptor of that point.

As you may have deduced already, when two points lie on a flat surface, the sphere radius will be infinite. If, on the other hand, they lie on the curved face of a cylinder, the radius will be more or less the same as that of the cylinder. This allows us to tell objects apart with RSD. The algorithm takes a parameter that sets the maximum radius at which the points will be considered to be part of a plane.

<center><gallery widths=300px>
File:RSD_sphere.png | Sphere that fits two points with their normals (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
File:RSD_values.png | RSD values for a cloud; points on a flat surface have maximum values (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
</gallery></center>

This is the code for compiling the RSD descriptor:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/rsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RSD descriptors for each point.
pcl::PointCloud<pcl::PrincipalRadiiRSD>::Ptr descriptors(new pcl::PointCloud<pcl::PrincipalRadiiRSD>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// RSD estimation object.
RSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::PrincipalRadiiRSD> rsd;
rsd.setInputCloud(cloud);
rsd.setInputNormals(normals);
rsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
rsd.setRadiusSearch(0.05);
// Plane radius. Any radius larger than this is considered infinite (a plane).
rsd.setPlaneRadius(0.1);
// Do we want to save the full distance-angle histograms?
rsd.setSaveHistograms(false);

rsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

Optionally, you can use the <span style="color:#FF1493">"setSaveHistograms()"</span> function to enable the saving of the full distance-angle histograms, and then use <span style="color:#FF1493">"getHistograms()"</span> to retrieve them.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Maximum Radius, [Subdivisions], [Save full histograms]
* '''Output''': RSD descriptors
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.391.7398&rep=rep1&type=pdf General 3D Modelling of Novel Objects from a Single View] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_s_d_estimation.html pcl::RSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RSD.tar.gz Download]
</div>

==3DSC==

The 3D Shape Context is a descriptor that extends its existing 2D counterpart to the third dimension. It works by creating a support structure (a sphere, to be precise) centered at the point we are computing the descriptor for, with the given search radius. The "north pole" of that sphere (the notion of "up") is pointed to match the normal at that point. Then, the sphere is divided in 3D regions or bins. In the first 2 coordinates (azimuth and elevation) the divisions are equally spaced, but in the third (the radial dimension), divisions are logarithmically spaced so they are smaller towards the center. A minimum radius can be specified to prevent very small bins, that would be too sensitive to small changes in the surface.

[[Image:3DSC_support_structure.png|thumb|center|200px|Support structure to compute the 3DSC for a point (image from [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf original paper]).]]

For each bin, a weighted count is accumulated for every neighboring point that lies within. The weight depends on the volume of the bin and the local point density (number of points around the current neighbor). This gives the descriptor some degree of resolution invariance.

We have mentioned that the sphere is given the direction of the normal. This still leaves one degree of freedom (only two axes have been locked, the azimuth remains free). Because of this, the descriptor so far does not cope with rotation. To overcome this (so the same point in two different clouds has the same value), the support sphere is rotated around the normal ''N'' times (a number of degrees that corresponds with the divisions in the azimuth) and the process is repeated for each, giving a total of ''N'' descriptors for that point.

You can compute the 3DSC descriptor the following way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/3dsc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the 3DSC descriptors for each point.
pcl::PointCloud<pcl::ShapeContext1980>::Ptr descriptors(new pcl::PointCloud<pcl::ShapeContext1980>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// 3DSC estimation object.
pcl::ShapeContext3DEstimation<pcl::PointXYZ, pcl::Normal, pcl::ShapeContext1980> sc3d;
sc3d.setInputCloud(cloud);
sc3d.setInputNormals(normals);
sc3d.setSearchMethod(kdtree);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
sc3d.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
sc3d.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
sc3d.setPointDensityRadius(0.05 / 5.0);

sc3d.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Minimal radius, Point density radius
* '''Output''': 3DSC descriptors
* '''Publication''':
** [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf Recognizing Objects in Range Data Using Regional Point Descriptors] (Andrea Frome et al., 2004)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_shape_context3_d_estimation.html pcl::ShapeContext3DEstimation]
* [http://robotica.unileon.es/~victorm/PCL_3DSC.tar.gz Download]
</div>

===USC===

The Unique Shape Context descriptor extends the 3DSC by defining a local reference frame, in order to provide an unique orientation for each point. This not only improves the accuracy of the descriptor, it also reduces its size, as computing multiple descriptors to account for orientation is no longer necessary.

You can check the second publication listed below to learn more about how the LRF is computed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/usc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the USC descriptors for each point.
pcl::PointCloud<pcl::UniqueShapeContext1960>::Ptr descriptors(new pcl::PointCloud<pcl::UniqueShapeContext1960>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// USC estimation object.
pcl::UniqueShapeContext<pcl::PointXYZ, pcl::UniqueShapeContext1960, pcl::ReferenceFrame> usc;
usc.setInputCloud(cloud);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
usc.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
usc.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
usc.setPointDensityRadius(0.05 / 5.0);
// Set the radius to compute the Local Reference Frame.
usc.setLocalRadius(0.05);

usc.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk). For 1.7 and below, change UniqueShapeContext1960 to ShapeContext1980, and edit CMakeLists.txt.'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Minimal radius, Point density radius, Local radius
* '''Output''': USC descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/3dor10.pdf Unique Shape Context for 3D Data Description] (Federico Tombari et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_unique_shape_context.html pcl::UniqueShapeContext]
* [http://robotica.unileon.es/~victorm/PCL_USC.tar.gz Download]
</div>

==SHOT==

SHOT stands for Signature of Histograms of Orientations. Like 3DSC, it encodes information about the topology (surface) withing a spherical support structure. This sphere is divided in 32 bins or volumes, with 8 divisions along the azimuth, 2 along the elevation, and 2 along the radius. For every volume, a one-dimensional local histogram is computed. The variable chosen is the angle between the normal of the keypoint and the current point within that volume (to be precise, the cosine, which was found to be better suitable).

[[Image:SHOT_support_structure.png|thumb|center|200px|Support structure to compute SHOT. Only 4 azimuth divisions are shown for clarity (image from [http://vision.deis.unibo.it/fede/papers/eccv10.pdf original paper]).]]

When all local histograms have been computed, they are stitched together in a final descriptor. Like the USC descriptor, SHOT makes use of a local reference frame, making it rotation invariant. It is also robust to noise and clutter.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the SHOT descriptors for each point.
pcl::PointCloud<pcl::SHOT352>::Ptr descriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// SHOT estimation object.
pcl::SHOTEstimation<pcl::PointXYZ, pcl::Normal, pcl::SHOT352> shot;
shot.setInputCloud(cloud);
shot.setInputNormals(normals);
// The radius that defines which of the keypoint's neighbors are described.
// If too large, there may be clutter, and if too small, not enough points may be found.
shot.setRadiusSearch(0.02);

shot.compute(*descriptors);
}</syntaxhighlight>

Like with FPFH, a multithreading-optimized variant is available with <span style="color:#FF1493">"SHOTEstimationOMP"</span>, that makes use of OpenMP. You need to include the header <span style="color:#FF1493">"shot_omp.h"</span>. Also, another variant that uses the texture for matching is available, <span style="color:#FF1493">"SHOTColorEstimation"</span>, with an optimized version too (see the second publication for more details). It outputs a <span style="color:#FF1493">"SHOT1344"</span> descriptor.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius
* '''Output''': SHOT descriptors
* '''Publications''':
** [http://vision.deis.unibo.it/fede/papers/eccv10.pdf Unique Signatures of Histograms for Local Surface Description] (Federico Tombari et al., 2010)
** [http://www.vision.deis.unibo.it/fede/papers/icip11.pdf A Combined Texture-Shaped Descriptor for Enhanced 3D Feature Matching] (Federico Tombari et al., 2011)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation.html pcl::SHOTEstimation],
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation.html pcl::SHOTColorEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation_o_m_p.html pcl::SHOTEstimationOMP]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation_o_m_p.html pcl::SHOTColorEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_SHOT.tar.gz Download]
</div>

==Spin image==

The Spin Image (SI) is the oldest descriptor we are going to see here. It has been around since 1997, but it still sees some use for certain applications. It was originally designed to describe surfaces made by vertices, edges and polygons, but it has been since adapted for point clouds. The descriptor is unlike all others in that the output resembles an image that can be compared with another with the usual means.

The support structure used is a cylinder, centered at the point, with a given radius and height, and aligned with the normal. This cylinder is divided radially and vertically into volumes. For each one, the number of neighbors lying inside is added up, eventually producing a descriptor. Weighting and interpolation are used to improve the result. The final descriptor can be seen as a grayscale image where dark areas correspond to volumes with higher point density.

[[Image:Spin images.png|thumb|center|400px|Spin images computed for 3 points of a model (image from [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf original thesis]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/spin_image.h>

// A handy typedef.
typedef pcl::Histogram<153> SpinImage;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the spin image for each point.
pcl::PointCloud<SpinImage>::Ptr descriptors(new pcl::PointCloud<SpinImage>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Spin image estimation object.
pcl::SpinImageEstimation<pcl::PointXYZ, pcl::Normal, SpinImage> si;
si.setInputCloud(cloud);
si.setInputNormals(normals);
// Radius of the support cylinder.
si.setRadiusSearch(0.02);
// Set the resolution of the spin image (the number of bins along one dimension).
// Note: you must change the output histogram size to reflect this.
si.setImageWidth(8);

si.compute(*descriptors);
}</syntaxhighlight>

The Spin Image estimation object provides more methods for tuning the estimation, so checking the API is recommended.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius, Image resolution
* '''Output''': Spin images
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf Spin-Images: A Representation for 3-D Surface Matching] (Andrew Edie Johnson, 1997)
** [http://www.cs.jhu.edu/~misha/Papers/Johnson99.pdf Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes] (Andrew Edie Johnson and Martial Hebert, 1999)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_spin_image_estimation.html pcl::SpinImageEstimation]
* [http://robotica.unileon.es/~victorm/PCL_spin_image.tar.gz Download]
</div>

==RIFT==

The Rotation-Invariant Feature Transform, like the spin image, takes some concepts from 2D features, in this case from the Scale-Invariant Feature Transform ([https://en.wikipedia.org/wiki/Scale-invariant_feature_transform SIFT]). It is the only descriptor seen here that requires intensity information in order to compute it (it can be obtained from the RGB color values). This means, of course, that you will not be able to use RIFT with standard XYZ clouds, you also need the texture.

In the first step, a circular patch (with the given radius) is fitted on the surface the point lies on. This patch is divided into concentric rings, according to the chosen distance bin size. Then, an histogram is populated with all the point's neighbors lying inside a sphere centered at that point and with the mentioned radius. The distance and the orientation of the intensity gradient at each point are considered. To make it rotation invariant, the angle between the gradient orientation and the vector pointing outward from the center of the patch is measured.

[[Image:RIFT.png|thumb|center|600px|RIFT feature values at 3 different locations in the descriptor (image from [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf original paper]).]]

The authors' original implementation uses 4 rings and 8 histogram orientations, which produce a descriptor of size 32. RIFT is not robust to texture flipping, though this was never considered a big issue.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/intensity_gradient.h>
#include <pcl/features/rift.h>

// A handy typedef.
typedef pcl::Histogram<32> RIFT32;

int
main(int argc, char** argv)
{
// Object for storing the point cloud with color information.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloudColor(new pcl::PointCloud<pcl::PointXYZRGB>);
// Object for storing the point cloud with intensity value.
pcl::PointCloud<pcl::PointXYZI>::Ptr cloudIntensity(new pcl::PointCloud<pcl::PointXYZI>);
// Object for storing the intensity gradients.
pcl::PointCloud<pcl::IntensityGradient>::Ptr gradients(new pcl::PointCloud<pcl::IntensityGradient>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RIFT descriptor for each point.
pcl::PointCloud<RIFT32>::Ptr descriptors(new pcl::PointCloud<RIFT32>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloudColor) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Convert the RGB to intensity.
pcl::PointCloudXYZRGBtoXYZI(*cloudColor, *cloudIntensity);

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZI, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudIntensity);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZI>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZI>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Compute the intensity gradients.
pcl::IntensityGradientEstimation < pcl::PointXYZI, pcl::Normal, pcl::IntensityGradient,
pcl::common::IntensityFieldAccessor<pcl::PointXYZI> > ge;
ge.setInputCloud(cloudIntensity);
ge.setInputNormals(normals);
ge.setRadiusSearch(0.03);
ge.compute(*gradients);

// RIFT estimation object.
pcl::RIFTEstimation<pcl::PointXYZI, pcl::IntensityGradient, RIFT32> rift;
rift.setInputCloud(cloudIntensity);
rift.setSearchMethod(kdtree);
// Set the intensity gradients to use.
rift.setInputGradient(gradients);
// Radius, to get all neighbors within.
rift.setRadiusSearch(0.02);
// Set the number of bins to use in the distance dimension.
rift.setNrDistanceBins(4);
// Set the number of bins to use in the gradient orientation dimension.
rift.setNrGradientBins(8);
// Note: you must change the output histogram size to reflect the previous values.

rift.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Intensity gradients, Radius, Distance bins, Gradient bins
* '''Output''': RIFT descriptors
* '''Publication''':
** [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf A Sparse Texture Representation Using Local Affine Regions] (Svetlana Lazebnik et al., 2005)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_r_i_f_t_estimation.html pcl::RIFTEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_intensity_gradient_estimation.html pcl::IntensityGradientEstimation]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_intensity_gradient.html pcl::IntensityGradient]
* [http://robotica.unileon.es/~victorm/PCL_RIFT.tar.gz Download]
</div>

==NARF==

The Normal Aligned Radial Feature is the only descriptor here that does not take a point cloud as input. Instead, it works with range images. A range image is a common RGB image in which the distance to the point that corresponds to a certain pixel is encoded as a color value in the [https://en.wikipedia.org/wiki/Visible_spectrum visible light spectrum]: the points that are closer to the camera would be violet, while the points near the maximum sensor range would be red.

NARF also requires us to find suitable keypoints to compute the descriptor for. NARF keypoints are located near an object's corners, and this also requires to find the borders (transitions from foreground to background), which are trivial to find with a range image. Because of this lengthy pipeline, I will describe the whole process in different sections.

===Obtaining a range image===

Because we always work with point clouds, I will now explain how you can convert one into a range image, in order to use it for the NARF descriptor. PCL provides a couple of handy classes to perform the conversion, given that you fill the camera data correctly.

A range image can be created in two ways. First, we can use spherical projection, which would give us an image similar to the ones produced by a LIDAR sensor. Second, we can use planar projection, which is better suitable for camera-like sensors as the Kinect or the Xtion, and will not have the characteristic distortion of the first one.

====Spherical projection====

The following code will take a point cloud and create a range image from it, using spherical projection:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the range image object:

// Angular resolution is the angular distance between pixels.
// Kinect: 57° horizontal FOV, 43° vertical FOV, 640x480 (chosen here).
// Xtion: 58° horizontal FOV, 45° vertical FOV, 640x480.
float angularResolutionX = (float)(57.0f / 640.0f * (M_PI / 180.0f));
float angularResolutionY = (float)(43.0f / 480.0f * (M_PI / 180.0f));
// Maximum horizontal and vertical angles. For example, for a full panoramic scan,
// the first would be 360º. Choosing values that adjust to the real sensor will
// decrease the time it takes, but don't worry. If the values are bigger than
// the real ones, the image will be automatically cropped to discard empty zones.
float maxAngleX = (float)(60.0f * (M_PI / 180.0f));
float maxAngleY = (float)(50.0f * (M_PI / 180.0f));
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;
// Border size. If greater than 0, a border of "unobserved" points will be left
// in the image when it is cropped.
int borderSize = 1;

// Range image object.
pcl::RangeImage rangeImage;
rangeImage.createFromPointCloud(*cloud, angularResolutionX, angularResolutionY,
maxAngleX, maxAngleY, sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange, borderSize);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Range image");
viewer.showRangeImage(rangeImage);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

Here you can see an example of the output range image:

[[Image:Range_image_spherical.png|thumb|center|400px|Range image of a point cloud, using spherical projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Angular resolution, Maximum angle, Sensor pose, Coordinate frame, Noise level, Maximum range, Border size
* '''Output''': Range image (spherical projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image.html pcl::RangeImage]
* [http://robotica.unileon.es/~victorm/PCL_range_image_spherical.tar.gz Download]
</div>

====Planar projection====

As mentioned, planar projection will give better results with clouds taken from a depth camera:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the planar range image object:

// Image size. Both Kinect and Xtion work at 640x480.
int imageSizeX = 640;
int imageSizeY = 480;
// Center of projection. here, we choose the middle of the image.
float centerX = 640.0f / 2.0f;
float centerY = 480.0f / 2.0f;
// Focal length. The value seen here has been taken from the original depth images.
// It is safe to use the same value vertically and horizontally.
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;

// Planar range image object.
pcl::RangeImagePlanar rangeImagePlanar;
rangeImagePlanar.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Planar range image");
viewer.showRangeImage(rangeImagePlanar);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_planar.png|thumb|center|400px|Range image of a point cloud, using planar projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Image size, Projection center, Focal length, Sensor pose, Coordinate frame, Noise level, Maximum range
* '''Output''': Range image (planar projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_planar.html pcl::RangeImagePlanar]
* [http://robotica.unileon.es/~victorm/PCL_range_image_planar.tar.gz Download]
</div>

If you prefer to do the conversion in real time while you inspect the cloud, PCL ships with an [https://github.com/PointCloudLibrary/pcl/tree/master/doc/tutorials/content/sources/openni_range_image_visualization example] that fetches an <span style="color:#FF1493">"openni_wrapper::DepthImage"</span> from an OpenNI device and creates the range image from it. You can adapt the code of the [[PCL/OpenNI_tutorial_1:_Installing_and_testing#Testing | example]] example from tutorial 1 to save it to disk with the function [http://docs.pointclouds.org/trunk/group__io.html#ga7291a029cdcde32ca3639d07dc6491b9 pcl::io::saveRangeImagePlanarFilePNG()].

===Extracting borders===

NARF keypoints are located near the edges of objects in the range image, so in order to find them, we first have to extract the borders. A border is defined as an abrupt change from foreground to background. In a range image, this can be easily seen because there is a "jump" in the depth value of two adjacent pixels.

[[Image:Range_image_border_detection.png|thumb|center|400px|Border detection on a range image (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

There are three types of borders. ''Object borders'' consist of the pixels (or points) located on the very edge of an object (the outermost points still belonging to the object). ''Shadow borders'' are points in the background on the edge of occlusions (empty areas in the background due to the objects in front covering them). Notice that, when the cloud is seen from the sensor's viewpoint, object and shadow points will seem adjacent. Finally, ''veil points'' are points interpolated between the previous two which appear in scans taken with LIDAR sensors, so we do not have to worry about them here.

<center><gallery widths=300px>
File:Range_image_border_type.png | Types of borders (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:Range_image_borders_example.png | Example of object and shadow borders on a cloud.
</gallery></center>

The algorithm basically compares a point's depth with the values of its neighbors, and if a big difference is found, we know it is due to a border. Points closer to the sensor will be marked as object borders, and the other ones as shadow borders.

PCL provides a class for extracting borders of a range image:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the borders.
pcl::PointCloud<pcl::BorderDescription>::Ptr borders(new pcl::PointCloud<pcl::BorderDescription>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Border extractor object.
pcl::RangeImageBorderExtractor borderExtractor(&rangeImage);

borderExtractor.compute(*borders);

// Visualize the borders.
pcl::visualization::RangeImageVisualizer* viewer = NULL;
viewer = pcl::visualization::RangeImageVisualizer::getRangeImageBordersWidget(rangeImage,
-std::numeric_limits<float>::infinity(),
std::numeric_limits<float>::infinity(),
false, *borders, "Borders");

while (!viewer->wasStopped())
{
viewer->spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_borders.png|thumb|center|400px|Borders found on the range image.]]

You can use the extractor's <span style="color:#FF1493">"getParameters()"</span> function to get a [http://docs.pointclouds.org/trunk/structpcl_1_1_range_image_border_extractor_1_1_parameters.html pcl::RangeImageBorderExtractor::Parameters] struct with the settings that will be used.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image
* '''Output''': Borders
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/range_image_border_extraction.php How to extract borders from range images]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_border_extractor.html pcl::RangeImageBorderExtractor]
* [http://robotica.unileon.es/~victorm/PCL_range_image_borders.tar.gz Download]
</div>

===Finding keypoints===

Citing the [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original publication]:''

''"We have the following requirements for our interest point extraction procedure:

# ''It must take information about borders and the surface structure into account.''
# ''It must select positions that can be reliably detected even if the object is observed from another perspective.''
# ''The points must be on positions that provide stable areas for normal estimation or the descriptor calculation in general."''

The procedure is the following: for every point in the range image, a score is computed that conveys how much the surface changes in its neighborhood (this is tuned with the ''support size'' σ, which is the diameter of the sphere used to find neighboring points). Also, the dominant direction of this change is computed. Then, this direction is compared with those of the neighbors, trying to find how stable the point is (if the directions are very different, that means the point is not stable, and that the surface around changes a lot). Points that are near the object's corners (but not exactly on the very edge) will be good keypoints, yet stable enough.

<center><gallery widths=300px>
File:NARF_keypoints.png | NARF keypoints (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:NARF_keypoints_support_sizes.png | Interest regions with a support size of 20cm (up) and 1m (down) (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
</gallery></center>

In PCL, NARF keypoints can be found this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

pcl::RangeImageBorderExtractor borderExtractor;
// Keypoint detection object.
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
// The support size influences how big the surface of interest will be,
// when finding keypoints from the border information.
detector.getParameters().support_size = 0.2f;

detector.compute(*keypoints);

// Visualize the keypoints.
pcl::visualization::RangeImageVisualizer viewer("NARF keypoints");
viewer.showRangeImage(rangeImage);
for (size_t i = 0; i < keypoints->points.size(); ++i)
{
viewer.markPoint(keypoints->points[i] % rangeImage.width,
keypoints->points[i] / rangeImage.width,
// Set the color of the pixel to red (the background
// circle is already that color). All other parameters
// are left untouched, check the API for more options.
pcl::visualization::Vector3ub(1.0f, 0.0f, 0.0f));
}

while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_NARF_keypoints.png|thumb|center|400px|NARF keypoints found on the range image.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Border extractor, Support Size
* '''Output''': NARF keypoints
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_keypoint_extraction.php How to extract NARF keypoint from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_keypoint.html pcl::NarfKeypoint]
* [http://robotica.unileon.es/~victorm/PCL_NARF_keypoints.tar.gz Download]
</div>

===Computing the descriptor===

We have created the range image from a point cloud, and we have extracted the borders in order to find good keypoints. Now it is time to compute the NARF descriptor for each keypoint.

The NARF descriptor encodes information about surface changes around a point. First, a local range patch is created around the point. It is like a small range image centered at that point, aligned with the normal (it would seem as if we were looking at the point along the normal). Then, a star pattern with ''n'' beams is overlaid onto the patch, also centered at the point. For every beam, a value is computed, that reflects how much the surface under it changes. The stronger the change is, and the closer to the center it is, the higher the final value will be. The ''n'' resulting values compose the final descriptor.

[[Image:NARF_descriptor.png|thumb|center|600px|Computing the NARF descriptor for a keypoint (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

The descriptor right now is not invariant to rotations around the normal. To achieve this, the whole possible 360 degrees are binned into an histogram. The value of each bin is computed from the descriptor values according to the angle. Then, the bin with the highest value is considered the dominant orientation, and the descriptor is shifted according to it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/features/narf_descriptor.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);
// Object for storing the NARF descriptors.
pcl::PointCloud<pcl::Narf36>::Ptr descriptors(new pcl::PointCloud<pcl::Narf36>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Extract the keypoints.
pcl::RangeImageBorderExtractor borderExtractor;
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
detector.getParameters().support_size = 0.2f;
detector.compute(*keypoints);

// The NARF estimator needs the indices in a vector, not a cloud.
std::vector<int> keypoints2;
keypoints2.resize(keypoints->points.size());
for (unsigned int i = 0; i < keypoints->size(); ++i)
keypoints2[i] = keypoints->points[i];
// NARF estimation object.
pcl::NarfDescriptor narf(&rangeImage, &keypoints2);
// Support size: choose the same value you used for keypoint extraction.
narf.getParameters().support_size = 0.2f;
// If true, the rotation invariant version of NARF will be used. The histogram
// will be shifted according to the dominant orientation to provide robustness to
// rotations around the normal.
narf.getParameters().rotation_invariant = true;

narf.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Key points, Support Size
* '''Output''': NARF descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_feature_extraction.php How to extract NARF Features from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_descriptor.html pcl::NarfDescriptor]
* [http://robotica.unileon.es/~victorm/PCL_NARF_descriptor.tar.gz Download]
</div>

==RoPS==

The Rotational Projection Statistics (RoPS) feature is a bit different from the other descriptors because it works with a triangle mesh, so a previous triangulation step is needed for generating this mesh from the cloud. Apart from that, most concepts are similar.

In order to compute RoPS for a keypoint, the local surface is cropped according to a support radius, so only points and triangles lying inside are taken into account. Then, a local reference frame (LRF) is computed, giving the descriptor its rotational invariance. A coordinate system is created with the point as the origin, and the axes aligned with the LRF. Then, for every axis, several steps are performed.

First, the local surface is rotated around the current axis. The angle is determined by one of the parameters, which sets the number of rotations. Then, all points in the local surface are projected onto the XY, XZ and YZ planes. For each one, statistical information about the distribution of the projected points is computed, and concatenated to form the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/surface/gp3.h>
#include <pcl/features/rops_estimation.h>

// A handy typedef.
typedef pcl::Histogram<135> ROPS135;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing both the points and the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr cloudNormals(new pcl::PointCloud<pcl::PointNormal>);
// Object for storing the ROPS descriptor for each point.
pcl::PointCloud<ROPS135>::Ptr descriptors(new pcl::PointCloud<ROPS135>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Perform triangulation.
pcl::concatenateFields(*cloud, *normals, *cloudNormals);
pcl::search::KdTree<pcl::PointNormal>::Ptr kdtree2(new pcl::search::KdTree<pcl::PointNormal>);
kdtree2->setInputCloud(cloudNormals);
pcl::GreedyProjectionTriangulation<pcl::PointNormal> triangulation;
pcl::PolygonMesh triangles;
triangulation.setSearchRadius(0.025);
triangulation.setMu(2.5);
triangulation.setMaximumNearestNeighbors(100);
triangulation.setMaximumSurfaceAngle(M_PI / 4); // 45 degrees.
triangulation.setNormalConsistency(false);
triangulation.setMinimumAngle(M_PI / 18); // 10 degrees.
triangulation.setMaximumAngle(2 * M_PI / 3); // 120 degrees.
triangulation.setInputCloud(cloudNormals);
triangulation.setSearchMethod(kdtree2);
triangulation.reconstruct(triangles);

// Note: you should only compute descriptors for chosen keypoints. It has
// been omitted here for simplicity.

// RoPs estimation object.
pcl::ROPSEstimation<pcl::PointXYZ, ROPS135> rops;
rops.setInputCloud(cloud);
rops.setSearchMethod(kdtree);
rops.setRadiusSearch(0.03);
rops.setTriangles(triangles.polygons);
// Number of partition bins that is used for distribution matrix calculation.
rops.setNumberOfPartitionBins(5);
// The greater the number of rotations is, the bigger the resulting descriptor.
// Make sure to change the histogram size accordingly.
rops.setNumberOfRotations(3);
// Support radius that is used to crop the local surface of the point.
rops.setSupportRadius(0.025);

rops.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Triangles, Search method, Support radius, Number of rotations, Number of partition bins
* '''Output''': RoPS descriptors
* '''Publication''':
** [http://arxiv.org/pdf/1304.3192v1.pdf Rotational Projection Statistics for 3D Local Surface Description and Object Recognition] (Yulan Guo et al., 2013)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_o_p_s_estimation.html pcl::ROPSEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RoPS.tar.gz Download]
</div>

=Global descriptors=

Global descriptors encode object geometry. They are not computed for individual points, but for a whole cluster that represents an object. Because of this, a preprocessing step ([[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]]) is required, in order to retrieve possible candidates.

Global descriptors are used for object recognition and classification, geometric analysis (object type, shape...), and pose estimation.

You should also know that many local descriptors can also be used as global ones. This can be done with descriptors that use a radius to search for neighbors (as PFH does). The trick is to compute it for one single point in the object cluster, and set the radius to the maximum possible distance between any two points (so all points in the cluster are considered as neighbors).

==VFH==

The Viewpoint Feature Histogram is based on the FPFH. Because the latter is invariant to the object's pose, the authors decided to expand it by including information about the viewpoint. Also, the FPFH is estimated once for the whole cluster, not for every point.

The VFH is made up by two parts: a viewpoint direction component, and an extended FPFH component. To compute the first one, the object's [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]] is found, which is the point that results from averaging the X, Y and Z coordinates of all points. Then, the vector between the viewpoint (the position of the sensor) and this centroid is computed, and normalized. Finally, for all points in the cluster, the angle between this vector and their normal is calculated, and the result is binned into an histogram. The vector is translated to each point when computing the angle because it makes the descriptor scale invariant.

The second component is computed like the FPFH (that results in 3 histograms for the 3 angular features, α, φ and θ), with some differences: it is only computed for the centroid, using the computed viewpoint direction vector as its normal (as the point, obviously, does not have a normal), and setting all the cluster's points as neighbors.

<center><gallery widths=300px>
File:VFH_viewpoint_component.png | Viewpoint component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
File:VFH_extended_FPFH_component.png | Extended FPFH component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
</gallery></center>

The resulting 4 histograms (1 for the viewpoint component, 3 for the extended FPFH component) are concatenated to build the final VFH descriptor. By default, the bins are normalized using the total number of points in the cluster. This makes the VFH descriptor invariant to scale.

[[Image:VFH_histogram.png|thumb|center|400px| VFH histogram (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).]]

The PCL implementation computes an additional fifth histogram with the distances of the cluster points to the centroid (the Shape Distribution Component, SDC), increading the size of the output descriptor from 263 to 308. The SDC is taken from the CVFH descriptor that we will see in the next section, and makes the result more robust.

The VFH of an already clustered object can be computed this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the VFH descriptor.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// VFH estimation object.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
// Optionally, we can normalize the bins of the resulting histogram,
// using the total number of points.
vfh.setNormalizeBins(true);
// Also, we can normalize the SDC with the maximum size found between
// the centroid and any of the cluster's points.
vfh.setNormalizeDistance(false);

vfh.compute(*descriptor);
}</syntaxhighlight>

Because only one VFH descriptor is computed for the whole cluster, the size of the cloud object that stores the result will be 1.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, [Normalize bins], [Normalize SDC]
* '''Output''': VFH descriptor
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_estimation.php Estimating VFH signatures for a set of points]
* '''Publication''':
** [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf Fast 3D Recognition and Pose Using the Viewpoint Feature Histogram] (Radu Bogdan Rusu et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_v_f_h_estimation.html pcl::VFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_VFH.tar.gz Download]
</div>

===CVFH===

The original VFH descriptor is not robust to occlusion or other sensor artifacts, or measurement errors. If the object cluster is missing many points, the resulting computed centroid will differ from the original, altering the final descriptor, and preventing a positive match from being found. Because of that, the Clustered Viewpoint Feature Histogram (CVFH) was introduced.

The idea is very simple: instead of computing a single VFH histogram for the whole cluster, the object is first divided in stable, smooth regions using [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Region_growing | region-growing segmentation]], that enforces several constraints in the distances and differences of normals of the points belonging to every region. Then, a VFH is computed for every region. Thanks to this, an object can be found in a scene, as long as at least one of its regions is fully visible.

<center><gallery widths=300px>
File:CVFH_occlusion.png | Typical occlusion issues in a point cloud (image from original paper).
File:CVFH_regions.png | Object regions computed for the CVFH (image from original paper).
</gallery></center>

Additionally, a Shape Distribution Component (SDC) is also computed and included. It encodes information about the distribution of the points arond the region's centroid, measuring the distances. The SDC allows to differentiate objects with similar characteristics (size and normal distribution), like two planar surfaces from each other.

The authors proposed to discard the histogram normalization step that is performed in VFH. This has the effect of making the descriptor dependant of scale, so an object of a certain size would not match a bigger or smaller copy of itself. It also makes CVFH more robust to occlusion.

CVFH is invariant to the camera roll angle, like most global descriptors. This is so because rotations about that camera axis do not change the observable geometry that descriptors are computed from, limiting the pose estimation to 5 DoF. The use of a Camera Roll Histogram (CRH) has been proposed to overcome this.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CVFH estimation object.
pcl::CVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> cvfh;
cvfh.setInputCloud(object);
cvfh.setInputNormals(normals);
cvfh.setSearchMethod(kdtree);
// Set the maximum allowable deviation of the normals,
// for the region segmentation step.
cvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
// Set the curvature threshold (maximum disparity between curvatures),
// for the region segmentation step.
cvfh.setCurvatureThreshold(1.0);
// Set to true to normalize the bins of the resulting histogram,
// using the total number of points. Note: enabling it will make CVFH
// invariant to scale just like VFH, but the authors encourage the opposite.
cvfh.setNormalizeBins(false);

cvfh.compute(*descriptors);
}</syntaxhighlight>

You can further customize the segmentation step with <span style="color:#FF1493">"setClusterTolerance()"</span> (to set the maximum Euclidean distance between points in the same cluster) and <span style="color:#FF1493">"setMinPoints()"</span>. The size of the output will be equal to the number of regions the object was divided in. Also, check the functions <span style="color:#FF1493">"getCentroidClusters()"</span> and <span style="color:#FF1493">"getCentroidNormalClusters()"</span>, you can use them to get information about the centroids used to compute the different CVFH descriptors.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points]
* '''Output''': CVFH descriptors
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_v_f_h_estimation.html pcl::CVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CVFH.tar.gz Download]
</div>

====OUR-CVFH====

The Oriented, Unique and Repeatable CVFH expands the previous descriptor, adding the computation of an unique reference frame to make it more robust.

OUR-CVFH relies on the use of Semi-Global Unique Reference Frames (SGURFs), which are repeatable coordinate systems computed for each region. Not only they remove the invariance to camera roll and allow to extract the 6DoF pose directly without additional steps, but they also improve the spatial descriptiveness.

The first part of the computation is akin to CVFH, but after segmentation, the points in each region are filtered once more according to the difference between their normals and the region's average normal. This results in better shaped regions, improving the estimation of the Reference Frames (RFs).

After this, the SGURF is computed for each region. Disambiguation is performed to decide the sign of the axes, according to the points' distribution. If this is not enough and the sign remains ambiguous, multiple RFs will need to be created to account for it. Finally, the OUR-CVFH descriptor is computed. The original Shape Distribution Component (SDC) is discarded, and the surface is now described according to the RFs.

[[Image:OUR-CVFH.png|thumb|center|600px| SGURF frame and resulting histogram of a region (image from [http://vision.deis.unibo.it/fede/papers/dagm12.pdf original paper]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/our_cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the OUR-CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// OUR-CVFH estimation object.
pcl::OURCVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> ourcvfh;
ourcvfh.setInputCloud(object);
ourcvfh.setInputNormals(normals);
ourcvfh.setSearchMethod(kdtree);
ourcvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
ourcvfh.setCurvatureThreshold(1.0);
ourcvfh.setNormalizeBins(false);
// Set the minimum axis ratio between the SGURF axes. At the disambiguation phase,
// this will decide if additional Reference Frames need to be created, if ambiguous.
ourcvfh.setAxisRatio(0.8);

ourcvfh.compute(*descriptors);
}</syntaxhighlight>

You can use the <span style="color:#FF1493">"getTransforms()"</span> function to get the transformations aligning the cloud to the corresponding SGURF.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points], [Axis ratio]
* '''Output''': OUR-CVFH descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/dagm12.pdf OUR-CVFH – Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram for Object Recognition and 6DOF Pose Estimation] (Aitor Aldoma et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_o_u_r_c_v_f_h_estimation.html pcl::OURCVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_OUR-CVFH.tar.gz Download]
</div>

==ESF==

The Ensemble of Shape Functions (ESF) is a combination of 3 different shape functions that describe certain properties of the cloud's points: distances, angles and area. This descriptor is very unique because it does not require normal information. Actually, it does not need any preprocessing, as it is robust to noise and incomplete surfaces.

The algorithm uses a voxel grid as an approximation of the real surface. It iterates through all the points in the cloud: for every iteration, 3 random points are chosen. For these points, the shape functions are computed:

* '''D2''': this function computes the distances between point pairs (3 overall). Then, for every pair, it checks if the line that connects both points lies entirely inside the surface, entirely outside (crossing free space), or both. Depending on this, the distance value will be binned to one of three possible histograms: IN, OUT or MIXED.
* '''D2 ratio''': an additional histogram for the ratio between parts of the line inside the surface, and parts outside. This value will be 0 if the line is completely outside, 1 if completely inside, and some value in between if mixed.
* '''D3''': this computes the square root of the area of the triangle formed by the 3 points. Like D2, the result is also classified as IN, OUT or MIXED, each with its own histogram.
* '''A3''': this function computes the angle formed by the points. Then, the value is binned depending on how the line opposite to the angle is (once again, as IN, OUT or MIXED).

[[Image:ESF.png|thumb|center|600px| ESF descriptor (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

After the loop is over, we are left with 10 subhistograms (IN, OUT and MIXED for D2, D3 and A3, and an additional one for the ratio). Each one has 64 bins, so the size of the final ESF descriptor is 640.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the ESF descriptor.
pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// ESF estimation object.
pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
esf.setInputCloud(object);

esf.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster)
* '''Output''': ESF descriptor
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6181760 Ensemble of shape functions for 3D object classification] (requires IEEE Xplore subscription) (Walter Wohlkinger and Markus Vincze, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_e_s_f_estimation.html pcl::ESFEstimation]
* [http://robotica.unileon.es/~victorm/PCL_ESF.tar.gz Download]
</div>

==GFPFH==

As you may have guessed, GFPFH stands for Global Fast Point Feature Histogram, the global version of the FPFH descriptor. GFPFH was designed for the task of helping a robot navigate its environment, having some context of the objects around it.

The first step before being able to compute the descriptor is surface categorization. A set of logical primitives (the classes, or categories) is created, which depends on the type of objects we expect the robot to find on the scene. For example, if we know there will be a coffee mug, we create three: one for the handle, and the other two for the outer and inner faces. Then, FPFH descriptors are computed, and everything is fed to a [https://en.wikipedia.org/wiki/Conditional_random_field Conditional Random Field] (CRF) algorithm. The CRF will label each surface with one of the previous categories, so we end up with a cloud where each point has been classified depending of the type of object (or object's region) it belongs to.

[[Image:FPFH_CRF.png|thumb|center|300px| Classification of objects made with FPFH and CRF (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

Now, the GFPFH descriptor can be computed with the result of the classification step. It will encode what the object is made of, so the robot can easily recognize it. First, an [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Octree | octree]] is created, dividing the object in voxel leaves. For every leaf, a set of probabilities is created, one for each class. Each one stores the probability of that leaf belonging to the class, and it is computed according to the number of points in that leaf that have been labelled as that class, and the total number of points. Then, for every pair of leaves in the octree, a line is casted, connecting them. Every leaf in its path is checked for occupancy, storing the result in an histogram. If the leaf is empty (free space), a value of 0 is saved. Otherwise, the leaf probabilities are used.

[[Image:GFPFH.png|thumb|center|500px| Computing the GFPFH with a voxel grid (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

The following code will compute the GFPFH for a cloud with label information. The categorization step is up to you, as it depends largely on the type of the scene, and the use you are going to give it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/gfpfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZL>::Ptr object(new pcl::PointCloud<pcl::PointXYZL>);
// Object for storing the GFPFH descriptor.
pcl::PointCloud<pcl::GFPFHSignature16>::Ptr descriptor(new pcl::PointCloud<pcl::GFPFHSignature16>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZL>(argv[1], *object) != 0)
{
return -1;
}

// Note: you should now perform classification on the cloud's points. See the
// original paper for more details. For this example, we will now consider 4
// different classes, and randomly label each point as one of them.
for (size_t i = 0; i < object->points.size(); ++i)
{
object->points[i].label = 1 + i % 4;
}

// ESF estimation object.
pcl::GFPFHEstimation<pcl::PointXYZL, pcl::PointXYZL, pcl::GFPFHSignature16> gfpfh;
gfpfh.setInputCloud(object);
// Set the object that contains the labels for each point. Thanks to the
// PointXYZL type, we can use the same object we store the cloud in.
gfpfh.setInputLabels(object);
// Set the size of the octree leaves to 1cm (cubic).
gfpfh.setOctreeLeafSize(0.01);
// Set the number of classes the cloud has been labelled with (default is 16).
gfpfh.setNumberOfClasses(4);

gfpfh.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Labels, Number of classes, Leaf size
* '''Output''': GFPFH descriptor
* '''Publication''':
** [https://www.willowgarage.com/sites/default/files/iccv09.pdf Detecting and Segmenting Objects for Mobile Manipulation] (Radu Bogdan Rusu et al., 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_f_p_f_h_estimation.html pcl::GFPFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GFPFH.tar.gz Download]
</div>

==GRSD==

The global version of the Radius-based Surface Descriptor works in a similar fashion to GFPFH. A voxelization and a surface categorization step are performed beforehand, labelling all surface patches according to the geometric category (plane, cylinder, edge, rim, sphere), using RSD. Then, the whole cluster is classified into one of these categories, and the GRSD descriptor is computed from this.

[[Image:GRSD.png|thumb|center|400px| Classification of objects for GRSD and resulting histogram (image from [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf original paper]).]]

To compute it:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/grsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the GRSD descriptors for each point.
pcl::PointCloud<pcl::GRSDSignature21>::Ptr descriptors(new pcl::PointCloud<pcl::GRSDSignature21>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// GRSD estimation object.
GRSDEstimation<PointXYZ, Normal, GRSDSignature21> grsd;
grsd.setInputCloud(cloud);
grsd.setInputNormals(normals);
grsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
grsd.setRadiusSearch(0.05);

grsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Radius
* '''Output''': GRSD descriptor
* '''Publications''':
** [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
** [http://www.mi.t.u-tokyo.ac.jp/top/downloadpublication/36 Voxelized Shape and Color Histograms for RGB-D] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_r_s_d_estimation.html pcl::GRSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GRSD.tar.gz Download]
</div>

=Saving and loading=

You can save a descriptor to a file just [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Writing_to_file|like with any other cloud type]]. One caveat, though. If you are using a descriptor that has its own custom type, like <span style="color:#FF1493">"PFHSignature125"</span>, everything will be OK. But with descriptors that do not (where you have to use <span style="color:#FF1493">"pcl::Histogram<>"</span>), you will get this error: <span style="color:#FF1493">''"POINT_TYPE_NOT_PROPERLY_REGISTERED"''</span>. In order to save or load from a file, PCL's IO functions need to know about the number, type and size of the fields. To solve this, you will have to properly register a new point type for your descriptor. For example, this will work for the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#RoPS|RoPS]] descriptor example we saw earlier:

<syntaxhighlight lang=CPP>POINT_CLOUD_REGISTER_POINT_STRUCT(ROPS135,
(float[135], histogram, histogram)
)</syntaxhighlight>

Add the previous to your code (change it accordingly), and you will be able to save and load descriptors as usual.

=Visualization=

Sometimes it is desired to check a visual representation of a descriptor, perhaps to analyze the distribution of data over different bins. Because they are saved as histograms, this is something trivial to do. PCL offers a couple of classes to do this.

==PCLHistogramVisualizer==

<span style="color:#FF1493">"PCLHistogramVisualizer"</span> is the simplest way to plot an histogram. The class has little functionality, but it does its job. Only one call is necessary to give the histogram and its size:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/histogram_visualizer.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLHistogramVisualizer viewer;
// We need to set the size of the descriptor beforehand.
viewer.addFeatureHistogram(*descriptor, 308);

viewer.spin();
}</syntaxhighlight>

[[Image:Histogram_visualizer_VFH.png|thumb|center|500px| VFH histogram seen with the Histogram Visualizer.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_histogram_visualizer.html pcl::visualization::PCLHistogramVisualizer]
* [http://robotica.unileon.es/~victorm/PCL_histogram_visualizer.tar.gz Download]
</div>

==PCLPlotter==

This class has all the methods from <span style="color:#FF1493">"PCLHistogramVisualizer"</span> (which will be deprecated soon) plus a lot more features. The code is almost the same:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/pcl_plotter.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLPlotter plotter;
// We need to set the size of the descriptor beforehand.
plotter.addFeatureHistogram(*descriptor, 308);

plotter.plot();
}</syntaxhighlight>

[[Image:PCL_plotter_VFH.png|thumb|center|500px| VFH histogram seen with the PCLPlotter class.]]

If you have raw data (such as a vector of floats) you can use the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html#aaf23b2b1c2f91c517cfde387ee1b654e addHistogramData()] function to plot it as an histogram.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pcl_plotter.php PCLPlotter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html pcl::visualization::PCLPlotter]
* [http://robotica.unileon.es/~victorm/PCL_plotter.tar.gz Download]
</div>

==PCL Viewer==

This program, included with PCL, will also let you open and visualize a saved descriptor. Internally, it uses [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#PCLPlotter|PCLPlotter]]. You can invoke the viewer from the command line like this, so it comes handy:

<syntaxhighlight lang=Bash>pcl_viewer <descriptor_file></syntaxhighlight>

[[Image:PCL_viewer_VFH.png|thumb|center|500px| VFH histogram seen with ''pcl_viewer''.]]

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 4: 3D object recognition (descriptors)

2015-11-02T11:08:04Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

It is time to learn the basics of one of the most interesting applications of point cloud processing: 3D object recognition. Akin to 2D recognition, this technique relies on finding good keypoints (characteristic points) in the cloud, and matching them to a set of previously saved ones. But 3D has several advantages over 2D: namely, we will be able to estimate with decent accuracy the exact position and orientation of the object, relative to the sensor. Also, 3D object recognition tends to be more robust to clutter (crowded scenes where objects in the front occluding objects in the background). And finally, having information about the object's shape will help with collision avoidance or grasping operations.

In this first tutorial we will see what descriptors are, how many types are there available in PCL, and how to compute them.

=Overview=

The basis of 3D object recognition is to find a set of correspondences between two different clouds, one of them containing the object we are looking for. In order to do this, we need a way to compare points in an unambiguous manner. Until now, we have worked with points that store the XYZ coordinates, the RGB color... but none of those properties are unique enough. In two sequential scans, two points could share the same coordinates despite belonging to different surfaces, and using the color information takes us back to 2D recognition, will all the lightning related problems.

In a previous tutorial, we talked about [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Feature_estimation | features]], before introducing the normals. Normals are an example of feature, because they encode information about the vicinity of the point. That is, the neighboring points are taken into account when computing it, giving us an idea of how the surrounding surface is. But this is not enough. For a feature to be optimal, it must meet the following criteria:

# '''It must be robust to transformations''': rigid transformations (the ones that do not change the distance between points) like translations and rotations must not affect the feature. Even if we play with the cloud a bit beforehand, there should be no difference.
# '''It must be robust to noise''': measurement errors that cause noise should not change the feature estimation much.
# '''It must be resolution invariant''': if sampled with different density (like after performing downsampling), the result must be identical or similar.

This is where descriptors come in. They are more complex (and precise) signatures of a point, that encode a lot of information about the surrounding geometry. The purpose is to unequivocally identify a point across multiple point clouds, no matter the noise, resolution or transformations. Also, some of them capture additional data about the object they belong to, like the viewpoint (that lets us retrieve the pose).

[[Image:Correspondence.jpg|thumb|center|600px|Finding correspondences between point features of two clouds (image from http://pointclouds.org/).]]

There are many 3D descriptors implemented into PCL. Each one has its own method for computing unique values for a point. Some use the difference between the angles of the normals of the point and its neighbors, for example. Others use the distances between the points. Because of this, some are inherently better or worse for certain purposes. A given descriptor may be scale invariant, and another one may be better with occlusions and partial views of objects. Which one you choose depends on what you want to do.

After calculating the necessary values, an additional step is performed to reduce the descriptor size: the result is binned into an [https://en.wikipedia.org/wiki/Histogram histogram]. To do this, the value range of each variable that makes up the descriptor is divided into ''n'' subdivisions, and the number of occurrences in each one is counted. Try to imagine a descriptor that computes a single variable, that ranges from 1 to 100. We choose to create 10 bins for it, so the first bin would gather all occurrences between 1 and 10, the second from 11 to 20, and so on. We look at the value of the variable for the first point-neighbor pair, and it is 27, so we increment the value of the third bin by 1. We keep doing this until we get a final histogram for that keypoint. The bin size must be carefully chosen depending on how descriptive that variable is (the variables do not have to share the same number of bins, and also the bins do not have to be of the same size; if for example most values from the previous example fell in the 50-100 range then it would be sensible to have more bins of smaller size in that range).

Descriptors can be classified in two main categories: global and local. The process for computing and using each one (recognition pipeline) is different, so each will be explained in its own section in this article.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/how_features_work.php How 3D Features work in PCL]
** [https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features Overview and Comparison of Features]
** [http://www.pointclouds.org/assets/icra2013/pcl_features_icra13.pdf How does a good feature look like?]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

==Table==

The following table will give you a hint of how many descriptors are there in PCL, and some of their features:

{| class="wikitable sortable" style="margin: auto; text-align:center" cellpadding="15"
|+ 3D descriptors
|- style="vertical-align:middle;"
! scope="col"| Name
! scope="col"| Type
! scope="col"| Size<sup>†</sup>
! scope="col"| Custom PointType<sup>††</sup>
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#PFH|PFH]] (Point Feature Histogram)
| style="color: sienna;" | Local
| 125
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#FPFH|FPFH]] (Fast Point Feature Histogram)
| style="color: sienna;" | Local
| 33
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RSD|RSD]] (Radius-Based Surface Descriptor)
| style="color: sienna;" | Local
| 289
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#3DSC|3DSC]] (3D Shape Context)
| style="color: sienna;" | Local
| 1980
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#USC|USC]] (Unique Shape Context)
| style="color: sienna;" | Local
| 1960
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#SHOT|SHOT]] (Signatures of Histograms of Orientations)
| style="color: sienna;" | Local
| 352
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Spin_image|Spin image]]
| style="color: sienna;" | Local
| 153*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RIFT|RIFT]] (Rotation-Invariant Feature Transform)
| style="color: sienna;" | Local
| 32*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#NARF|NARF]] (Normal Aligned Radial Feature)
| style="color: sienna;" | Local
| 36
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RoPS|RoPS]] (Rotational Projection Statistics)
| style="color: sienna;" | Local
| 135*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#VFH|VFH]] (Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#CVFH|CVFH]] (Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#OUR-CVFH|OUR-CVFH]] (Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#ESF|ESF]] (Ensemble of Shape Functions)
| style="color: darkcyan;" | Global
| 640
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GFPFH|GFPFH]] (Global Fast Point Feature Histogram)
| style="color: darkcyan;" | Global
| 16
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GRSD|GRSD]] (Global Radius-Based Surface Descriptor)
| style="color: darkcyan;" | Global
| 21
| style="color: green;" | Yes
|}

† Values marked with an asterisk (*) indicate that the descriptor's size depends on some parameter(s), and the one given is for the default values.

†† Descriptors without their own custom PointType use the generic <span style="color:#FF1493">"pcl::Histogram<>"</span> type. See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|Saving and loading]].

Optionally, you can download a document with a less simple version of the table. Page format is A4, landscape:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Downloads''':
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.odt Table as original ODT] (uses font embedding, requires LibreOffice 4)
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.pdf Table as PDF]
</div>

=Local descriptors=

Local descriptors are computed for individual points that we give as input. They have no notion of what an object is, they just describe how the local geometry is around that point. Usually, it is your task to choose which points you want a descriptor to be computed for: the ''keypoints''. Most of the time, you can get away by just performing a downsampling and choosing all remaining points, but keypoint detectors are available, like the one used for [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF]], or [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#ISS|ISS]].

Local descriptors are used for object recognition and [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration | registration]]. Now we will see which ones are implemented into PCL.

==PFH==

PFH stands for Point Feature Histogram. It is one of the most important descriptors offered by PCL and the basis of others such as FPFH. The PFH tries to capture information of the geometry surrounding the point by analyzing the difference between the directions of the normals in the vicinity (and because of this, an imprecise normal estimation may produce low-quality descriptors).

First, the algorithm pairs all points in the vicinity (not just the chosen keypoint with its neighbors, but also the neighbors with themselves). Then, for each pair, a [https://en.wikipedia.org/wiki/Darboux_frame fixed coordinate frame] is computed from their normals. With this frame, the difference between the normals can be encoded with 3 angular variables. These variables, together with the euclidean distance between the points, are saved, and then binned to an histogram when all pairs have been computed. The final descriptor is the concatenation of the histograms of each variable (4 in total).

<center><gallery widths=300px>
File:PFH_neighbors.png | Point pairs established when computing the PFH for a point (image from http://pointclouds.org).
File:PFH_frame.png | Fixed coordinate frame and angular features computed for one of the pairs (image from http://pointclouds.org).
</gallery></center>

Computing descriptors in PCL is very easy, and the PFH is not an exception:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/pfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the PFH descriptors for each point.
pcl::PointCloud<pcl::PFHSignature125>::Ptr descriptors(new pcl::PointCloud<pcl::PFHSignature125>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// PFH estimation object.
pcl::PFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::PFHSignature125> pfh;
pfh.setInputCloud(cloud);
pfh.setInputNormals(normals);
pfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
pfh.setRadiusSearch(0.05);

pfh.compute(*descriptors);
}</syntaxhighlight>

As you can see, PCL uses the <span style="color:#FF1493">"PFHSignature125"</span> type to save the descriptor to. This means that the descriptor's size is 125 (the dimensionality of the feature vector). Dividing a feature in ''D'' dimensional space in ''B'' divisions requires a total of B<sup>D</sup> bins. The original proposal makes use of the distance between the points, but the implementation of PCL does not, as it was not considered discriminative enough (specially in 2.5D scans, where the distance between points increases the further away from the sensor). Hence, the remaining features (with one dimension each) can be divided in 5 divisions, resulting in a 125-dimensional (5<sup>3</sup>) vector.

The final object that stores the computed descriptors can be handled like a normal cloud (even saved to, or read from, disk), and it has the same number of "points" than the original one. The <span style="color:#FF1493">"PFHSignature125"</span> object at position 0, for example, stores the PFH descriptor for the <span style="color:#FF1493">"PointXYZ"</span> point at the same position in the cloud.

For additional details about the descriptor, check the original publications listed below, or PCL's tutorial.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': PFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pfh_estimation.php Point Feature Histograms (PFH) descriptors]
* '''Publications''':
** [http://ias.in.tum.de/_media/spezial/bib/rusu08ias.pdf Persistent Point Feature Histograms for 3D Point Clouds] (Radu Bogdan Rusu et al., 2008)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.4, page 50)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_p_f_h_estimation.html pcl::PFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_PFH.tar.gz Download]
</div>

===FPFH===

PFH gives accurate results, but it has a drawback: it is too computationally expensive to perform at real time. For a cloud of ''n'' keypoints with ''k'' neighbors considered, it has a [https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations complexity] of ''O(nk<sup>2</sup>)''. Because of this, a derived descriptor was created, named FPFH (Fast Point Feature Histogram).

The FPFH considers only the direct connections between the current keypoint and its neighbors, removing additional links between neighbors. This takes the complexity down to ''O(nk)''. Because of this, the resulting histogram is referred to as SPFH (Simplified Point Feature Histogram). The reference frame and the angular variables are computed as always.

[[Image:FPFH_neighbors.png|thumb|center|400px|Point pairs established when computing the FPFH for a point (image from http://pointclouds.org).]]

To account for the loss of these extra connections, an additional step takes place after all histograms have been computed: the SPFHs of a point's neighbors are "merged" with its own, weighted according to the distance. This has the effect of giving a point surface information of points as far away as 2 times the radius used. Finally, the 3 histograms (distance is not used) are concatenated to compose the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/fpfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the FPFH descriptors for each point.
pcl::PointCloud<pcl::FPFHSignature33>::Ptr descriptors(new pcl::PointCloud<pcl::FPFHSignature33>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// FPFH estimation object.
pcl::FPFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::FPFHSignature33> fpfh;
fpfh.setInputCloud(cloud);
fpfh.setInputNormals(normals);
fpfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
fpfh.setRadiusSearch(0.05);

fpfh.compute(*descriptors);
}</syntaxhighlight>

An additional implementation of the FPFH estimation that takes advantage of multithreaded optimizations (with the [https://en.wikipedia.org/wiki/OpenMP OpenMP API]) is available in the class <span style="color:#FF1493">"FPFHEstimationOMP"</span>. Its interface is identical to the standard unoptimized implementation. Using it will result in a big performance boost on multi-core systems, meaning faster computation times. Remember to include the header <span style="color:#FF1493">"fpfh_omp.h"</span> instead.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': FPFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/fpfh_estimation.php Fast Point Feature Histograms (FPFH) descriptors]
* '''Publications''':
** [http://files.rbrusu.com/publications/Rusu09ICRA.pdf Fast Point Feature Histograms (FPFH) for 3D Registration] (Radu Bogdan Rusu et al., 2009)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.5, page 57)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation.html pcl::FPFHEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation_o_m_p.html pcl::FPFHEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_FPFH.tar.gz Download]
</div>

==RSD==

The Radius-Based Surface Descriptor encodes the radial relationship of the point and its neighborhood. For every pair of the keypoint with a neighbor, the algorithm computes the distance between them, and the difference between their normals. Then, by assuming that both points lie on the surface of a sphere, said sphere is found by fitting not only the points, but also the normals (otherwise, there would be infinite possible spheres). Finally, from all the point-neighbor spheres, only the ones with the maximum and minimum radii are kept and saved to the descriptor of that point.

As you may have deduced already, when two points lie on a flat surface, the sphere radius will be infinite. If, on the other hand, they lie on the curved face of a cylinder, the radius will be more or less the same as that of the cylinder. This allows us to tell objects apart with RSD. The algorithm takes a parameter that sets the maximum radius at which the points will be considered to be part of a plane.

<center><gallery widths=300px>
File:RSD_sphere.png | Sphere that fits two points with their normals (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
File:RSD_values.png | RSD values for a cloud; points on a flat surface have maximum values (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
</gallery></center>

This is the code for compiling the RSD descriptor:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/rsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RSD descriptors for each point.
pcl::PointCloud<pcl::PrincipalRadiiRSD>::Ptr descriptors(new pcl::PointCloud<pcl::PrincipalRadiiRSD>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// RSD estimation object.
RSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::PrincipalRadiiRSD> rsd;
rsd.setInputCloud(cloud);
rsd.setInputNormals(normals);
rsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
rsd.setRadiusSearch(0.05);
// Plane radius. Any radius larger than this is considered infinite (a plane).
rsd.setPlaneRadius(0.1);
// Do we want to save the full distance-angle histograms?
rsd.setSaveHistograms(false);

rsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

Optionally, you can use the <span style="color:#FF1493">"setSaveHistograms()"</span> function to enable the saving of the full distance-angle histograms, and then use <span style="color:#FF1493">"getHistograms()"</span> to retrieve them.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Maximum Radius, [Subdivisions], [Save full histograms]
* '''Output''': RSD descriptors
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.391.7398&rep=rep1&type=pdf General 3D Modelling of Novel Objects from a Single View] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_s_d_estimation.html pcl::RSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RSD.tar.gz Download]
</div>

==3DSC==

The 3D Shape Context is a descriptor that extends its existing 2D counterpart to the third dimension. It works by creating a support structure (a sphere, to be precise) centered at the point we are computing the descriptor for, with the given search radius. The "north pole" of that sphere (the notion of "up") is pointed to match the normal at that point. Then, the sphere is divided in 3D regions or bins. In the first 2 coordinates (azimuth and elevation) the divisions are equally spaced, but in the third (the radial dimension), divisions are logarithmically spaced so they are smaller towards the center. A minimum radius can be specified to prevent very small bins, that would be too sensitive to small changes in the surface.

[[Image:3DSC_support_structure.png|thumb|center|200px|Support structure to compute the 3DSC for a point (image from [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf original paper]).]]

For each bin, a weighted count is accumulated for every neighboring point that lies within. The weight depends on the volume of the bin and the local point density (number of points around the current neighbor). This gives the descriptor some degree of resolution invariance.

We have mentioned that the sphere is given the direction of the normal. This still leaves one degree of freedom (only two axes have been locked, the azimuth remains free). Because of this, the descriptor so far does not cope with rotation. To overcome this (so the same point in two different clouds has the same value), the support sphere is rotated around the normal ''N'' times (a number of degrees that corresponds with the divisions in the azimuth) and the process is repeated for each, giving a total of ''N'' descriptors for that point.

You can compute the 3DSC descriptor the following way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/3dsc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the 3DSC descriptors for each point.
pcl::PointCloud<pcl::ShapeContext1980>::Ptr descriptors(new pcl::PointCloud<pcl::ShapeContext1980>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// 3DSC estimation object.
pcl::ShapeContext3DEstimation<pcl::PointXYZ, pcl::Normal, pcl::ShapeContext1980> sc3d;
sc3d.setInputCloud(cloud);
sc3d.setInputNormals(normals);
sc3d.setSearchMethod(kdtree);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
sc3d.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
sc3d.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
sc3d.setPointDensityRadius(0.05 / 5.0);

sc3d.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Minimal radius, Point density radius
* '''Output''': 3DSC descriptors
* '''Publication''':
** [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf Recognizing Objects in Range Data Using Regional Point Descriptors] (Andrea Frome et al., 2004)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_shape_context3_d_estimation.html pcl::ShapeContext3DEstimation]
* [http://robotica.unileon.es/~victorm/PCL_3DSC.tar.gz Download]
</div>

===USC===

The Unique Shape Context descriptor extends the 3DSC by defining a local reference frame, in order to provide an unique orientation for each point. This not only improves the accuracy of the descriptor, it also reduces its size, as computing multiple descriptors to account for orientation is no longer necessary.

You can check the second publication listed below to learn more about how the LRF is computed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/usc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the USC descriptors for each point.
pcl::PointCloud<pcl::UniqueShapeContext1960>::Ptr descriptors(new pcl::PointCloud<pcl::UniqueShapeContext1960>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// USC estimation object.
pcl::UniqueShapeContext<pcl::PointXYZ, pcl::UniqueShapeContext1960, pcl::ReferenceFrame> usc;
usc.setInputCloud(cloud);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
usc.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
usc.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
usc.setPointDensityRadius(0.05 / 5.0);
// Set the radius to compute the Local Reference Frame.
usc.setLocalRadius(0.05);

usc.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk). For 1.7 and below, change UniqueShapeContext1960 to ShapeContext1980, and edit CMakeLists.txt.'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Minimal radius, Point density radius, Local radius
* '''Output''': USC descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/3dor10.pdf Unique Shape Context for 3D Data Description] (Federico Tombari et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_unique_shape_context.html pcl::UniqueShapeContext]
* [http://robotica.unileon.es/~victorm/PCL_USC.tar.gz Download]
</div>

==SHOT==

SHOT stands for Signature of Histograms of Orientations. Like 3DSC, it encodes information about the topology (surface) withing a spherical support structure. This sphere is divided in 32 bins or volumes, with 8 divisions along the azimuth, 2 along the elevation, and 2 along the radius. For every volume, a one-dimensional local histogram is computed. The variable chosen is the angle between the normal of the keypoint and the current point within that volume (to be precise, the cosine, which was found to be better suitable).

[[Image:SHOT_support_structure.png|thumb|center|200px|Support structure to compute SHOT. Only 4 azimuth divisions are shown for clarity (image from [http://vision.deis.unibo.it/fede/papers/eccv10.pdf original paper]).]]

When all local histograms have been computed, they are stitched together in a final descriptor. Like the USC descriptor, SHOT makes use of a local reference frame, making it rotation invariant. It is also robust to noise and clutter.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the SHOT descriptors for each point.
pcl::PointCloud<pcl::SHOT352>::Ptr descriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// SHOT estimation object.
pcl::SHOTEstimation<pcl::PointXYZ, pcl::Normal, pcl::SHOT352> shot;
shot.setInputCloud(cloud);
shot.setInputNormals(normals);
// The radius that defines which of the keypoint's neighbors are described.
// If too large, there may be clutter, and if too small, not enough points may be found.
shot.setRadiusSearch(0.02);

shot.compute(*descriptors);
}</syntaxhighlight>

Like with FPFH, a multithreading-optimized variant is available with <span style="color:#FF1493">"SHOTEstimationOMP"</span>, that makes use of OpenMP. You need to include the header <span style="color:#FF1493">"shot_omp.h"</span>. Also, another variant that uses the texture for matching is available, <span style="color:#FF1493">"SHOTColorEstimation"</span>, with an optimized version too (see the second publication for more details). It outputs a <span style="color:#FF1493">"SHOT1344"</span> descriptor.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius
* '''Output''': SHOT descriptors
* '''Publications''':
** [http://vision.deis.unibo.it/fede/papers/eccv10.pdf Unique Signatures of Histograms for Local Surface Description] (Federico Tombari et al., 2010)
** [http://www.vision.deis.unibo.it/fede/papers/icip11.pdf A Combined Texture-Shaped Descriptor for Enhanced 3D Feature Matching] (Federico Tombari et al., 2011)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation.html pcl::SHOTEstimation],
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation.html pcl::SHOTColorEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation_o_m_p.html pcl::SHOTEstimationOMP]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation_o_m_p.html pcl::SHOTColorEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_SHOT.tar.gz Download]
</div>

==Spin image==

The Spin Image (SI) is the oldest descriptor we are going to see here. It has been around since 1997, but it still sees some use for certain applications. It was originally designed to describe surfaces made by vertices, edges and polygons, but it has been since adapted for point clouds. The descriptor is unlike all others in that the output resembles an image that can be compared with another with the usual means.

The support structure used is a cylinder, centered at the point, with a given radius and height, and aligned with the normal. This cylinder is divided radially and vertically into volumes. For each one, the number of neighbors lying inside is added up, eventually producing a descriptor. Weighting and interpolation are used to improve the result. The final descriptor can be seen as a grayscale image where dark areas correspond to volumes with higher point density.

[[Image:Spin images.png|thumb|center|400px|Spin images computed for 3 points of a model (image from [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf original thesis]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/spin_image.h>

// A handy typedef.
typedef pcl::Histogram<153> SpinImage;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the spin image for each point.
pcl::PointCloud<SpinImage>::Ptr descriptors(new pcl::PointCloud<SpinImage>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Spin image estimation object.
pcl::SpinImageEstimation<pcl::PointXYZ, pcl::Normal, SpinImage> si;
si.setInputCloud(cloud);
si.setInputNormals(normals);
// Radius of the support cylinder.
si.setRadiusSearch(0.02);
// Set the resolution of the spin image (the number of bins along one dimension).
// Note: you must change the output histogram size to reflect this.
si.setImageWidth(8);

si.compute(*descriptors);
}</syntaxhighlight>

The Spin Image estimation object provides more methods for tuning the estimation, so checking the API is recommended.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius, Image resolution
* '''Output''': Spin images
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf Spin-Images: A Representation for 3-D Surface Matching] (Andrew Edie Johnson, 1997)
** [http://www.cs.jhu.edu/~misha/Papers/Johnson99.pdf Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes] (Andrew Edie Johnson and Martial Hebert, 1999)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_spin_image_estimation.html pcl::SpinImageEstimation]
* [http://robotica.unileon.es/~victorm/PCL_spin_image.tar.gz Download]
</div>

==RIFT==

The Rotation-Invariant Feature Transform, like the spin image, takes some concepts from 2D features, in this case from the Scale-Invariant Feature Transform ([https://en.wikipedia.org/wiki/Scale-invariant_feature_transform SIFT]). It is the only descriptor seen here that requires intensity information in order to compute it (it can be obtained from the RGB color values). This means, of course, that you will not be able to use RIFT with standard XYZ clouds, you also need the texture.

In the first step, a circular patch (with the given radius) is fitted on the surface the point lies on. This patch is divided into concentric rings, according to the chosen distance bin size. Then, an histogram is populated with all the point's neighbors lying inside a sphere centered at that point and with the mentioned radius. The distance and the orientation of the intensity gradient at each point are considered. To make it rotation invariant, the angle between the gradient orientation and the vector pointing outward from the center of the patch is measured.

[[Image:RIFT.png|thumb|center|600px|RIFT feature values at 3 different locations in the descriptor (image from [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf original paper]).]]

The authors' original implementation uses 4 rings and 8 histogram orientations, which produce a descriptor of size 32. RIFT is not robust to texture flipping, though this was never considered a big issue.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/intensity_gradient.h>
#include <pcl/features/rift.h>

// A handy typedef.
typedef pcl::Histogram<32> RIFT32;

int
main(int argc, char** argv)
{
// Object for storing the point cloud with color information.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloudColor(new pcl::PointCloud<pcl::PointXYZRGB>);
// Object for storing the point cloud with intensity value.
pcl::PointCloud<pcl::PointXYZI>::Ptr cloudIntensity(new pcl::PointCloud<pcl::PointXYZI>);
// Object for storing the intensity gradients.
pcl::PointCloud<pcl::IntensityGradient>::Ptr gradients(new pcl::PointCloud<pcl::IntensityGradient>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RIFT descriptor for each point.
pcl::PointCloud<RIFT32>::Ptr descriptors(new pcl::PointCloud<RIFT32>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloudColor) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Convert the RGB to intensity.
pcl::PointCloudXYZRGBtoXYZI(*cloudColor, *cloudIntensity);

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZI, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudIntensity);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZI>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZI>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Compute the intensity gradients.
pcl::IntensityGradientEstimation < pcl::PointXYZI, pcl::Normal, pcl::IntensityGradient,
pcl::common::IntensityFieldAccessor<pcl::PointXYZI> > ge;
ge.setInputCloud(cloudIntensity);
ge.setInputNormals(normals);
ge.setRadiusSearch(0.03);
ge.compute(*gradients);

// RIFT estimation object.
pcl::RIFTEstimation<pcl::PointXYZI, pcl::IntensityGradient, RIFT32> rift;
rift.setInputCloud(cloudIntensity);
rift.setSearchMethod(kdtree);
// Set the intensity gradients to use.
rift.setInputGradient(gradients);
// Radius, to get all neighbors within.
rift.setRadiusSearch(0.02);
// Set the number of bins to use in the distance dimension.
rift.setNrDistanceBins(4);
// Set the number of bins to use in the gradient orientation dimension.
rift.setNrGradientBins(8);
// Note: you must change the output histogram size to reflect the previous values.

rift.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Intensity gradients, Radius, Distance bins, Gradient bins
* '''Output''': RIFT descriptors
* '''Publication''':
** [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf A Sparse Texture Representation Using Local Affine Regions] (Svetlana Lazebnik et al., 2005)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_r_i_f_t_estimation.html pcl::RIFTEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_intensity_gradient_estimation.html pcl::IntensityGradientEstimation]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_intensity_gradient.html pcl::IntensityGradient]
* [http://robotica.unileon.es/~victorm/PCL_RIFT.tar.gz Download]
</div>

==NARF==

The Normal Aligned Radial Feature is the only descriptor here that does not take a point cloud as input. Instead, it works with range images. A range image is a common RGB image in which the distance to the point that corresponds to a certain pixel is encoded as a color value in the [https://en.wikipedia.org/wiki/Visible_spectrum visible light spectrum]: the points that are closer to the camera would be violet, while the points near the maximum sensor range would be red.

NARF also requires us to find suitable keypoints to compute the descriptor for. NARF keypoints are located near an object's corners, and this also requires to find the borders (transitions from foreground to background), which are trivial to find with a range image. Because of this lengthy pipeline, I will describe the whole process in different sections.

===Obtaining a range image===

Because we always work with point clouds, I will now explain how you can convert one into a range image, in order to use it for the NARF descriptor. PCL provides a couple of handy classes to perform the conversion, given that you fill the camera data correctly.

A range image can be created in two ways. First, we can use spherical projection, which would give us an image similar to the ones produced by a LIDAR sensor. Second, we can use planar projection, which is better suitable for camera-like sensors as the Kinect or the Xtion, and will not have the characteristic distortion of the first one.

====Spherical projection====

The following code will take a point cloud and create a range image from it, using spherical projection:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the range image object:

// Angular resolution is the angular distance between pixels.
// Kinect: 57° horizontal FOV, 43° vertical FOV, 640x480 (chosen here).
// Xtion: 58° horizontal FOV, 45° vertical FOV, 640x480.
float angularResolutionX = (float)(57.0f / 640.0f * (M_PI / 180.0f));
float angularResolutionY = (float)(43.0f / 480.0f * (M_PI / 180.0f));
// Maximum horizontal and vertical angles. For example, for a full panoramic scan,
// the first would be 360º. Choosing values that adjust to the real sensor will
// decrease the time it takes, but don't worry. If the values are bigger than
// the real ones, the image will be automatically cropped to discard empty zones.
float maxAngleX = (float)(60.0f * (M_PI / 180.0f));
float maxAngleY = (float)(50.0f * (M_PI / 180.0f));
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;
// Border size. If greater than 0, a border of "unobserved" points will be left
// in the image when it is cropped.
int borderSize = 1;

// Range image object.
pcl::RangeImage rangeImage;
rangeImage.createFromPointCloud(*cloud, angularResolutionX, angularResolutionY,
maxAngleX, maxAngleY, sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange, borderSize);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Range image");
viewer.showRangeImage(rangeImage);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

Here you can see an example of the output range image:

[[Image:Range_image_spherical.png|thumb|center|400px|Range image of a point cloud, using spherical projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Angular resolution, Maximum angle, Sensor pose, Coordinate frame, Noise level, Maximum range, Border size
* '''Output''': Range image (spherical projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image.html pcl::RangeImage]
* [http://robotica.unileon.es/~victorm/PCL_range_image_spherical.tar.gz Download]
</div>

====Planar projection====

As mentioned, planar projection will give better results with clouds taken from a depth camera:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the planar range image object:

// Image size. Both Kinect and Xtion work at 640x480.
int imageSizeX = 640;
int imageSizeY = 480;
// Center of projection. here, we choose the middle of the image.
float centerX = 640.0f / 2.0f;
float centerY = 480.0f / 2.0f;
// Focal length. The value seen here has been taken from the original depth images.
// It is safe to use the same value vertically and horizontally.
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;

// Planar range image object.
pcl::RangeImagePlanar rangeImagePlanar;
rangeImagePlanar.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Planar range image");
viewer.showRangeImage(rangeImagePlanar);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_planar.png|thumb|center|400px|Range image of a point cloud, using planar projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Image size, Projection center, Focal length, Sensor pose, Coordinate frame, Noise level, Maximum range
* '''Output''': Range image (planar projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_planar.html pcl::RangeImagePlanar]
* [http://robotica.unileon.es/~victorm/PCL_range_image_planar.tar.gz Download]
</div>

If you prefer to do the conversion in real time while you inspect the cloud, PCL ships with an [https://github.com/PointCloudLibrary/pcl/tree/master/doc/tutorials/content/sources/openni_range_image_visualization example] that fetches an <span style="color:#FF1493">"openni_wrapper::DepthImage"</span> from an OpenNI device and creates the range image from it. You can adapt the code of the [[PCL/OpenNI_tutorial_1:_Installing_and_testing#Testing | example]] example from tutorial 1 to save it to disk with the function [http://docs.pointclouds.org/trunk/group__io.html#ga7291a029cdcde32ca3639d07dc6491b9 pcl::io::saveRangeImagePlanarFilePNG()].

===Extracting borders===

NARF keypoints are located near the edges of objects in the range image, so in order to find them, we first have to extract the borders. A border is defined as an abrupt change from foreground to background. In a range image, this can be easily seen because there is a "jump" in the depth value of two adjacent pixels.

[[Image:Range_image_border_detection.png|thumb|center|400px|Border detection on a range image (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

There are three types of borders. ''Object borders'' consist of the pixels (or points) located on the very edge of an object (the outermost points still belonging to the object). ''Shadow borders'' are points in the background on the edge of occlusions (empty areas in the background due to the objects in front covering them). Notice that, when the cloud is seen from the sensor's viewpoint, object and shadow points will seem adjacent. Finally, ''veil points'' are points interpolated between the previous two which appear in scans taken with LIDAR sensors, so we do not have to worry about them here.

<center><gallery widths=300px>
File:Range_image_border_type.png | Types of borders (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:Range_image_borders_example.png | Example of object and shadow borders on a cloud.
</gallery></center>

The algorithm basically compares a point's depth with the values of its neighbors, and if a big difference is found, we know it is due to a border. Points closer to the sensor will be marked as object borders, and the other ones as shadow borders.

PCL provides a class for extracting borders of a range image:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the borders.
pcl::PointCloud<pcl::BorderDescription>::Ptr borders(new pcl::PointCloud<pcl::BorderDescription>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Border extractor object.
pcl::RangeImageBorderExtractor borderExtractor(&rangeImage);

borderExtractor.compute(*borders);

// Visualize the borders.
pcl::visualization::RangeImageVisualizer* viewer = NULL;
viewer = pcl::visualization::RangeImageVisualizer::getRangeImageBordersWidget(rangeImage,
-std::numeric_limits<float>::infinity(),
std::numeric_limits<float>::infinity(),
false, *borders, "Borders");

while (!viewer->wasStopped())
{
viewer->spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_borders.png|thumb|center|400px|Borders found on the range image.]]

You can use the extractor's <span style="color:#FF1493">"getParameters()"</span> function to get a [http://docs.pointclouds.org/trunk/structpcl_1_1_range_image_border_extractor_1_1_parameters.html pcl::RangeImageBorderExtractor::Parameters] struct with the settings that will be used.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image
* '''Output''': Borders
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/range_image_border_extraction.php How to extract borders from range images]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_border_extractor.html pcl::RangeImageBorderExtractor]
* [http://robotica.unileon.es/~victorm/PCL_range_image_borders.tar.gz Download]
</div>

===Finding keypoints===

Citing the [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original publication]:''

''"We have the following requirements for our interest point extraction procedure:

# ''It must take information about borders and the surface structure into account.''
# ''It must select positions that can be reliably detected even if the object is observed from another perspective.''
# ''The points must be on positions that provide stable areas for normal estimation or the descriptor calculation in general."''

The procedure is the following: for every point in the range image, a score is computed that conveys how much the surface changes in its neighborhood (this is tuned with the ''support size'' σ, which is the diameter of the sphere used to find neighboring points). Also, the dominant direction of this change is computed. Then, this direction is compared with those of the neighbors, trying to find how stable the point is (if the directions are very different, that means the point is not stable, and that the surface around changes a lot). Points that are near the object's corners (but not exactly on the very edge) will be good keypoints, yet stable enough.

<center><gallery widths=300px>
File:NARF_keypoints.png | NARF keypoints (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:NARF_keypoints_support_sizes.png | Interest regions with a support size of 20cm (up) and 1m (down) (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
</gallery></center>

In PCL, NARF keypoints can be found this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

pcl::RangeImageBorderExtractor borderExtractor;
// Keypoint detection object.
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
// The support size influences how big the surface of interest will be,
// when finding keypoints from the border information.
detector.getParameters().support_size = 0.2f;

detector.compute(*keypoints);

// Visualize the keypoints.
pcl::visualization::RangeImageVisualizer viewer("NARF keypoints");
viewer.showRangeImage(rangeImage);
for (size_t i = 0; i < keypoints->points.size(); ++i)
{
viewer.markPoint(keypoints->points[i] % rangeImage.width,
keypoints->points[i] / rangeImage.width,
// Set the color of the pixel to red (the background
// circle is already that color). All other parameters
// are left untouched, check the API for more options.
pcl::visualization::Vector3ub(1.0f, 0.0f, 0.0f));
}

while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_NARF_keypoints.png|thumb|center|400px|NARF keypoints found on the range image.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Border extractor, Support Size
* '''Output''': NARF keypoints
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_keypoint_extraction.php How to extract NARF keypoint from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_keypoint.html pcl::NarfKeypoint]
* [http://robotica.unileon.es/~victorm/PCL_NARF_keypoints.tar.gz Download]
</div>

===Computing the descriptor===

We have created the range image from a point cloud, and we have extracted the borders in order to find good keypoints. Now it is time to compute the NARF descriptor for each keypoint.

The NARF descriptor encodes information about surface changes around a point. First, a local range patch is created around the point. It is like a small range image centered at that point, aligned with the normal (it would seem as if we were looking at the point along the normal). Then, a star pattern with ''n'' beams is overlaid onto the patch, also centered at the point. For every beam, a value is computed, that reflects how much the surface under it changes. The stronger the change is, and the closer to the center it is, the higher the final value will be. The ''n'' resulting values compose the final descriptor.

[[Image:NARF_descriptor.png|thumb|center|600px|Computing the NARF descriptor for a keypoint (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

The descriptor right now is not invariant to rotations around the normal. To achieve this, the whole possible 360 degrees are binned into an histogram. The value of each bin is computed from the descriptor values according to the angle. Then, the bin with the highest value is considered the dominant orientation, and the descriptor is shifted according to it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/features/narf_descriptor.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);
// Object for storing the NARF descriptors.
pcl::PointCloud<pcl::Narf36>::Ptr descriptors(new pcl::PointCloud<pcl::Narf36>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Extract the keypoints.
pcl::RangeImageBorderExtractor borderExtractor;
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
detector.getParameters().support_size = 0.2f;
detector.compute(*keypoints);

// The NARF estimator needs the indices in a vector, not a cloud.
std::vector<int> keypoints2;
keypoints2.resize(keypoints->points.size());
for (unsigned int i = 0; i < keypoints->size(); ++i)
keypoints2[i] = keypoints->points[i];
// NARF estimation object.
pcl::NarfDescriptor narf(&rangeImage, &keypoints2);
// Support size: choose the same value you used for keypoint extraction.
narf.getParameters().support_size = 0.2f;
// If true, the rotation invariant version of NARF will be used. The histogram
// will be shifted according to the dominant orientation to provide robustness to
// rotations around the normal.
narf.getParameters().rotation_invariant = true;

narf.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Key points, Support Size
* '''Output''': NARF descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_feature_extraction.php How to extract NARF Features from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_descriptor.html pcl::NarfDescriptor]
* [http://robotica.unileon.es/~victorm/PCL_NARF_descriptor.tar.gz Download]
</div>

==RoPS==

The Rotational Projection Statistics (RoPS) feature is a bit different from the other descriptors because it works with a triangle mesh, so a previous triangulation step is needed for generating this mesh from the cloud. Apart from that, most concepts are similar.

In order to compute RoPS for a keypoint, the local surface is cropped according to a support radius, so only points and triangles lying inside are taken into account. Then, a local reference frame (LRF) is computed, giving the descriptor its rotational invariance. A coordinate system is created with the point as the origin, and the axes aligned with the LRF. Then, for every axis, several steps are performed.

First, the local surface is rotated around the current axis. The angle is determined by one of the parameters, which sets the number of rotations. Then, all points in the local surface are projected onto the XY, XZ and YZ planes. For each one, statistical information about the distribution of the projected points is computed, and concatenated to form the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/surface/gp3.h>
#include <pcl/features/rops_estimation.h>

// A handy typedef.
typedef pcl::Histogram<135> ROPS135;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing both the points and the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr cloudNormals(new pcl::PointCloud<pcl::PointNormal>);
// Object for storing the ROPS descriptor for each point.
pcl::PointCloud<ROPS135>::Ptr descriptors(new pcl::PointCloud<ROPS135>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Perform triangulation.
pcl::concatenateFields(*cloud, *normals, *cloudNormals);
pcl::search::KdTree<pcl::PointNormal>::Ptr kdtree2(new pcl::search::KdTree<pcl::PointNormal>);
kdtree2->setInputCloud(cloudNormals);
pcl::GreedyProjectionTriangulation<pcl::PointNormal> triangulation;
pcl::PolygonMesh triangles;
triangulation.setSearchRadius(0.025);
triangulation.setMu(2.5);
triangulation.setMaximumNearestNeighbors(100);
triangulation.setMaximumSurfaceAngle(M_PI / 4); // 45 degrees.
triangulation.setNormalConsistency(false);
triangulation.setMinimumAngle(M_PI / 18); // 10 degrees.
triangulation.setMaximumAngle(2 * M_PI / 3); // 120 degrees.
triangulation.setInputCloud(cloudNormals);
triangulation.setSearchMethod(kdtree2);
triangulation.reconstruct(triangles);

// Note: you should only compute descriptors for chosen keypoints. It has
// been omitted here for simplicity.

// RoPs estimation object.
pcl::ROPSEstimation<pcl::PointXYZ, ROPS135> rops;
rops.setInputCloud(cloud);
rops.setSearchMethod(kdtree);
rops.setRadiusSearch(0.03);
rops.setTriangles(triangles.polygons);
// Number of partition bins that is used for distribution matrix calculation.
rops.setNumberOfPartitionBins(5);
// The greater the number of rotations is, the bigger the resulting descriptor.
// Make sure to change the histogram size accordingly.
rops.setNumberOfRotations(3);
// Support radius that is used to crop the local surface of the point.
rops.setSupportRadius(0.025);

rops.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Triangles, Search method, Support radius, Number of rotations, Number of partition bins
* '''Output''': RoPS descriptors
* '''Publication''':
** [http://arxiv.org/pdf/1304.3192v1.pdf Rotational Projection Statistics for 3D Local Surface Description and Object Recognition] (Yulan Guo et al., 2013)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_o_p_s_estimation.html pcl::ROPSEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RoPS.tar.gz Download]
</div>

=Global descriptors=

Global descriptors encode object geometry. They are not computed for individual points, but for a whole cluster that represents an object. Because of this, a preprocessing step ([[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]]) is required, in order to retrieve possible candidates.

Global descriptors are used for object recognition and classification, geometric analysis (object type, shape...), and pose estimation.

You should also know that many local descriptors can also be used as global ones. This can be done with descriptors that use a radius to search for neighbors (as PFH does). The trick is to compute it for one single point in the object cluster, and set the radius to the maximum possible distance between any two points (so all points in the cluster are considered as neighbors).

==VFH==

The Viewpoint Feature Histogram is based on the FPFH. Because the latter is invariant to the object's pose, the authors decided to expand it by including information about the viewpoint. Also, the FPFH is estimated once for the whole cluster, not for every point.

The VFH is made up by two parts: a viewpoint direction component, and an extended FPFH component. To compute the first one, the object's [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]] is found, which is the point that results from averaging the X, Y and Z coordinates of all points. Then, the vector between the viewpoint (the position of the sensor) and this centroid is computed, and normalized. Finally, for all points in the cluster, the angle between this vector and their normal is calculated, and the result is binned into an histogram. The vector is translated to each point when computing the angle because it makes the descriptor scale invariant.

The second component is computed like the FPFH (that results in 3 histograms for the 3 angular features, α, φ and θ), with some differences: it is only computed for the centroid, using the computed viewpoint direction vector as its normal (as the point, obviously, does not have a normal), and setting all the cluster's points as neighbors.

<center><gallery widths=300px>
File:VFH_viewpoint_component.png | Viewpoint component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
File:VFH_extended_FPFH_component.png | Extended FPFH component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
</gallery></center>

The resulting 4 histograms (1 for the viewpoint component, 3 for the extended FPFH component) are concatenated to build the final VFH descriptor. By default, the bins are normalized using the total number of points in the cluster. This makes the VFH descriptor invariant to scale.

[[Image:VFH_histogram.png|thumb|center|400px| VFH histogram (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).]]

The PCL implementation computes an additional fifth histogram with the distances of the cluster points to the centroid (the Shape Distribution Component, SDC), increading the size of the output descriptor from 263 to 308. The SDC is taken from the CVFH descriptor that we will see in the next section, and makes the result more robust.

The VFH of an already clustered object can be computed this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the VFH descriptor.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// VFH estimation object.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
// Optionally, we can normalize the bins of the resulting histogram,
// using the total number of points.
vfh.setNormalizeBins(true);
// Also, we can normalize the SDC with the maximum size found between
// the centroid and any of the cluster's points.
vfh.setNormalizeDistance(false);

vfh.compute(*descriptor);
}</syntaxhighlight>

Because only one VFH descriptor is computed for the whole cluster, the size of the cloud object that stores the result will be 1.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, [Normalize bins], [Normalize SDC]
* '''Output''': VFH descriptor
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_estimation.php Estimating VFH signatures for a set of points]
* '''Publication''':
** [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf Fast 3D Recognition and Pose Using the Viewpoint Feature Histogram] (Radu Bogdan Rusu et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_v_f_h_estimation.html pcl::VFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_VFH.tar.gz Download]
</div>

===CVFH===

The original VFH descriptor is not robust to occlusion or other sensor artifacts, or measurement errors. If the object cluster is missing many points, the resulting computed centroid will differ from the original, altering the final descriptor, and preventing a positive match from being found. Because of that, the Clustered Viewpoint Feature Histogram (CVFH) was introduced.

The idea is very simple: instead of computing a single VFH histogram for the whole cluster, the object is first divided in stable, smooth regions using [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Region_growing | region-growing segmentation]], that enforces several constraints in the distances and differences of normals of the points belonging to every region. Then, a VFH is computed for every region. Thanks to this, an object can be found in a scene, as long as at least one of its regions is fully visible.

<center><gallery widths=300px>
File:CVFH_occlusion.png | Typical occlusion issues in a point cloud (image from original paper).
File:CVFH_regions.png | Object regions computed for the CVFH (image from original paper).
</gallery></center>

Additionally, a Shape Distribution Component (SDC) is also computed and included. It encodes information about the distribution of the points arond the region's centroid, measuring the distances. The SDC allows to differentiate objects with similar characteristics (size and normal distribution), like two planar surfaces from each other.

The authors proposed to discard the histogram normalization step that is performed in VFH. This has the effect of making the descriptor dependant of scale, so an object of a certain size would not match a bigger or smaller copy of itself. It also makes CVFH more robust to occlusion.

CVFH is invariant to the camera roll angle, like most global descriptors. This is so because rotations about that camera axis do not change the observable geometry that descriptors are computed from, limiting the pose estimation to 5 DoF. The use of a Camera Roll Histogram (CRH) has been proposed to overcome this.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CVFH estimation object.
pcl::CVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> cvfh;
cvfh.setInputCloud(object);
cvfh.setInputNormals(normals);
cvfh.setSearchMethod(kdtree);
// Set the maximum allowable deviation of the normals,
// for the region segmentation step.
cvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
// Set the curvature threshold (maximum disparity between curvatures),
// for the region segmentation step.
cvfh.setCurvatureThreshold(1.0);
// Set to true to normalize the bins of the resulting histogram,
// using the total number of points. Note: enabling it will make CVFH
// invariant to scale just like VFH, but the authors encourage the opposite.
cvfh.setNormalizeBins(false);

cvfh.compute(*descriptors);
}</syntaxhighlight>

You can further customize the segmentation step with <span style="color:#FF1493">"setClusterTolerance()"</span> (to set the maximum Euclidean distance between points in the same cluster) and <span style="color:#FF1493">"setMinPoints()"</span>. The size of the output will be equal to the number of regions the object was divided in. Also, check the functions <span style="color:#FF1493">"getCentroidClusters()"</span> and <span style="color:#FF1493">"getCentroidNormalClusters()"</span>, you can use them to get information about the centroids used to compute the different CVFH descriptors.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points]
* '''Output''': CVFH descriptors
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_v_f_h_estimation.html pcl::CVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CVFH.tar.gz Download]
</div>

====OUR-CVFH====

The Oriented, Unique and Repeatable CVFH expands the previous descriptor, adding the computation of an unique reference frame to make it more robust.

OUR-CVFH relies on the use of Semi-Global Unique Reference Frames (SGURFs), which are repeatable coordinate systems computed for each region. Not only they remove the invariance to camera roll and allow to extract the 6DoF pose directly without additional steps, but they also improve the spatial descriptiveness.

The first part of the computation is akin to CVFH, but after segmentation, the points in each region are filtered once more according to the difference between their normals and the region's average normal. This results in better shaped regions, improving the estimation of the Reference Frames (RFs).

After this, the SGURF is computed for each region. Disambiguation is performed to decide the sign of the axes, according to the points' distribution. If this is not enough and the sign remains ambiguous, multiple RFs will need to be created to account for it. Finally, the OUR-CVFH descriptor is computed. The original Shape Distribution Component (SDC) is discarded, and the surface is now described according to the RFs.

[[Image:OUR-CVFH.png|thumb|center|600px| SGURF frame and resulting histogram of a region (image from [http://vision.deis.unibo.it/fede/papers/dagm12.pdf original paper]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/our_cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the OUR-CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// OUR-CVFH estimation object.
pcl::OURCVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> ourcvfh;
ourcvfh.setInputCloud(object);
ourcvfh.setInputNormals(normals);
ourcvfh.setSearchMethod(kdtree);
ourcvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
ourcvfh.setCurvatureThreshold(1.0);
ourcvfh.setNormalizeBins(false);
// Set the minimum axis ratio between the SGURF axes. At the disambiguation phase,
// this will decide if additional Reference Frames need to be created, if ambiguous.
ourcvfh.setAxisRatio(0.8);

ourcvfh.compute(*descriptors);
}</syntaxhighlight>

You can use the <span style="color:#FF1493">"getTransforms()"</span> function to get the transformations aligning the cloud to the corresponding SGURF.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points], [Axis ratio]
* '''Output''': OUR-CVFH descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/dagm12.pdf OUR-CVFH – Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram for Object Recognition and 6DOF Pose Estimation] (Aitor Aldoma et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_o_u_r_c_v_f_h_estimation.html pcl::OURCVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_OUR-CVFH.tar.gz Download]
</div>

==ESF==

The Ensemble of Shape Functions (ESF) is a combination of 3 different shape functions that describe certain properties of the cloud's points: distances, angles and area. This descriptor is very unique because it does not require normal information. Actually, it does not need any preprocessing, as it is robust to noise and incomplete surfaces.

The algorithm uses a voxel grid as an approximation of the real surface. It iterates through all the points in the cloud: for every iteration, 3 random points are chosen. For these points, the shape functions are computed:

* '''D2''': this function computes the distances between point pairs (3 overall). Then, for every pair, it checks if the line that connects both points lies entirely inside the surface, entirely outside (crossing free space), or both. Depending on this, the distance value will be binned to one of three possible histograms: IN, OUT or MIXED.
* '''D2 ratio''': an additional histogram for the ratio between parts of the line inside the surface, and parts outside. This value will be 0 if the line is completely outside, 1 if completely inside, and some value in between if mixed.
* '''D3''': this computes the square root of the area of the triangle formed by the 3 points. Like D2, the result is also classified as IN, OUT or MIXED, each with its own histogram.
* '''A3''': this function computes the angle formed by the points. Then, the value is binned depending on how the line opposite to the angle is (once again, as IN, OUT or MIXED).

[[Image:ESF.png|thumb|center|600px| ESF descriptor (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

After the loop is over, we are left with 10 subhistograms (IN, OUT and MIXED for D2, D3 and A3, and an additional one for the ratio). Each one has 64 bins, so the size of the final ESF descriptor is 640.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the ESF descriptor.
pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// ESF estimation object.
pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
esf.setInputCloud(object);

esf.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster)
* '''Output''': ESF descriptor
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6181760 Ensemble of shape functions for 3D object classification] (requires IEEE Xplore subscription) (Walter Wohlkinger and Markus Vincze, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_e_s_f_estimation.html pcl::ESFEstimation]
* [http://robotica.unileon.es/~victorm/PCL_ESF.tar.gz Download]
</div>

==GFPFH==

As you may have guessed, GFPFH stands for Global Fast Point Feature Histogram, the global version of the FPFH descriptor. GFPFH was designed for the task of helping a robot navigate its environment, having some context of the objects around it.

The first step before being able to compute the descriptor is surface categorization. A set of logical primitives (the classes, or categories) is created, which depends on the type of objects we expect the robot to find on the scene. For example, if we know there will be a coffee mug, we create three: one for the handle, and the other two for the outer and inner faces. Then, FPFH descriptors are computed, and everything is fed to a [https://en.wikipedia.org/wiki/Conditional_random_field Conditional Random Field] (CRF) algorithm. The CRF will label each surface with one of the previous categories, so we end up with a cloud where each point has been classified depending of the type of object (or object's region) it belongs to.

[[Image:FPFH_CRF.png|thumb|center|300px| Classification of objects made with FPFH and CRF (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

Now, the GFPFH descriptor can be computed with the result of the classification step. It will encode what the object is made of, so the robot can easily recognize it. First, an [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Octree | octree]] is created, dividing the object in voxel leaves. For every leaf, a set of probabilities is created, one for each class. Each one stores the probability of that leaf belonging to the class, and it is computed according to the number of points in that leaf that have been labelled as that class, and the total number of points. Then, for every pair of leaves in the octree, a line is casted, connecting them. Every leaf in its path is checked for occupancy, storing the result in an histogram. If the leaf is empty (free space), a value of 0 is saved. Otherwise, the leaf probabilities are used.

[[Image:GFPFH.png|thumb|center|500px| Computing the GFPFH with a voxel grid (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

The following code will compute the GFPFH for a cloud with label information. The categorization step is up to you, as it depends largely on the type of the scene, and the use you are going to give it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/gfpfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZL>::Ptr object(new pcl::PointCloud<pcl::PointXYZL>);
// Object for storing the GFPFH descriptor.
pcl::PointCloud<pcl::GFPFHSignature16>::Ptr descriptor(new pcl::PointCloud<pcl::GFPFHSignature16>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZL>(argv[1], *object) != 0)
{
return -1;
}

// Note: you should now perform classification on the cloud's points. See the
// original paper for more details. For this example, we will now consider 4
// different classes, and randomly label each point as one of them.
for (size_t i = 0; i < object->points.size(); ++i)
{
object->points[i].label = 1 + i % 4;
}

// ESF estimation object.
pcl::GFPFHEstimation<pcl::PointXYZL, pcl::PointXYZL, pcl::GFPFHSignature16> gfpfh;
gfpfh.setInputCloud(object);
// Set the object that contains the labels for each point. Thanks to the
// PointXYZL type, we can use the same object we store the cloud in.
gfpfh.setInputLabels(object);
// Set the size of the octree leaves to 1cm (cubic).
gfpfh.setOctreeLeafSize(0.01);
// Set the number of classes the cloud has been labelled with (default is 16).
gfpfh.setNumberOfClasses(4);

gfpfh.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Labels, Number of classes, Leaf size
* '''Output''': GFPFH descriptor
* '''Publication''':
** [https://www.willowgarage.com/sites/default/files/iccv09.pdf Detecting and Segmenting Objects for Mobile Manipulation] (Radu Bogdan Rusu et al., 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_f_p_f_h_estimation.html pcl::GFPFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GFPFH.tar.gz Download]
</div>

==GRSD==

The global version of the Radius-based Surface Descriptor works in a similar fashion to GFPFH. A voxelization and a surface categorization step are performed beforehand, labelling all surface patches according to the geometric category (plane, cylinder, edge, rim, sphere), using RSD. Then, the whole cluster is classified into one of these categories, and the GRSD descriptor is computed from this.

[[Image:GRSD.png|thumb|center|400px| Classification of objects for GRSD and resulting histogram (image from [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf original paper]).]]

To compute it:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/grsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the GRSD descriptors for each point.
pcl::PointCloud<pcl::GRSDSignature21>::Ptr descriptors(new pcl::PointCloud<pcl::GRSDSignature21>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// GRSD estimation object.
GRSDEstimation<PointXYZ, Normal, GRSDSignature21> grsd;
grsd.setInputCloud(cloud);
grsd.setInputNormals(normals);
grsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
grsd.setRadiusSearch(0.05);

grsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Radius
* '''Output''': GRSD descriptor
* '''Publications''':
** [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
** [http://www.mi.t.u-tokyo.ac.jp/top/downloadpublication/36 Voxelized Shape and Color Histograms for RGB-D] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_r_s_d_estimation.html pcl::GRSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GRSD.tar.gz Download]
</div>

=Saving and loading=

You can save a descriptor to a file just [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Writing_to_file|like with any other cloud type]]. One caveat, though. If you are using a descriptor that has its own custom type, like <span style="color:#FF1493">"PFHSignature125"</span>, everything will be OK. But with descriptors that do not (where you have to use <span style="color:#FF1493">"pcl::Histogram<>"</span>), you will get this error: <span style="color:#FF1493">''"POINT_TYPE_NOT_PROPERLY_REGISTERED"''</span>. In order to save or load from a file, PCL's IO functions need to know about the number, type and size of the fields. To solve this, you will have to properly register a new point type for your descriptor. For example, this will work for the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#RoPS|RoPS]] descriptor example we saw earlier:

<syntaxhighlight lang=CPP>POINT_CLOUD_REGISTER_POINT_STRUCT(ROPS135,
(float[135], histogram, histogram)
)</syntaxhighlight>

Add the previous to your code (change it accordingly), and you will be able to save and load descriptors as usual.

=Visualization=

Sometimes it is desired to check a visual representation of a descriptor, perhaps to analyze the distribution of data over different bins. Because they are saved as histograms, this is something trivial to do. PCL offers a couple of classes to do this.

==PCLHistogramVisualizer==

<span style="color:#FF1493">"PCLHistogramVisualizer"</span> is the simplest way to plot an histogram. The class has little functionality, but it does its job. Only one call is necessary to give the histogram and its size:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/histogram_visualizer.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLHistogramVisualizer viewer;
// We need to set the size of the descriptor beforehand.
viewer.addFeatureHistogram(*descriptor, 308);

viewer.spin();
}</syntaxhighlight>

[[Image:Histogram_visualizer_VFH.png|thumb|center|500px| VFH histogram seen with the Histogram Visualizer.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_histogram_visualizer.html pcl::visualization::PCLHistogramVisualizer]
* [http://robotica.unileon.es/~victorm/PCL_histogram_visualizer.tar.gz Download]
</div>

==PCLPlotter==

This class has all the methods from <span style="color:#FF1493">"PCLHistogramVisualizer"</span> (which will be deprecated soon) plus a lot more features. The code is almost the same:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/pcl_plotter.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLPlotter plotter;
// We need to set the size of the descriptor beforehand.
plotter.addFeatureHistogram(*descriptor, 308);

plotter.plot();
}</syntaxhighlight>

[[Image:PCL_plotter_VFH.png|thumb|center|500px| VFH histogram seen with the PCLPlotter class.]]

If you have raw data (such as a vector of floats) you can use the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html#aaf23b2b1c2f91c517cfde387ee1b654e addHistogramData()] function to plot it as an histogram.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pcl_plotter.php PCLPlotter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html pcl::visualization::PCLPlotter]
* [http://robotica.unileon.es/~victorm/PCL_plotter.tar.gz Download]
</div>

==PCL Viewer==

This program, included with PCL, will also let you open and visualize a saved descriptor. Internally, it uses [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#PCLPlotter|PCLPlotter]]. You can invoke the viewer from the command line like this, so it comes handy:

<syntaxhighlight lang=Bash>pcl_viewer <descriptor_file></syntaxhighlight>

[[Image:PCL_viewer_VFH.png|thumb|center|500px| VFH histogram seen with ''pcl_viewer''.]]

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-02T11:03:09Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [https://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [https://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [https://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [https://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [https://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [https://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [https://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://en.wikipedia.org/wiki/CUDA CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [https://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI troubleshooting

2015-11-01T23:29:07Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

This page tries to cover all of the issues I ran into while trying to get my Kinect device ready for developing with PCL. If you have any questions regarding this subject or know of a different bug, feel free to send me an email describing it, and I will add it to this section. Just remember to try Google first!

=CUDA=

==Unsupported configuration==

When you run the CUDA installer on a fairly modern Linux, you may get a message saying that you are trying to install it on an unsupported configuration. What this means is that CUDA can not work with the newest versions of the GCC compiler. As of version 7, the highest compatible GCC is 4.8.2 for Ubuntu 14.04 LTS and 4.9.1 for 14.10 (12.04 LTS has been deprecated). Check the compatibility table [http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#system-requirements here].

In order to solve this, you must install an older version, like 4.6 for Ubuntu 12.04, and set it as the "official" alternative for compiling (change 4.7 for your current version):

<syntaxhighlight lang=Bash>sudo apt-get install gcc-4.6 g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7</syntaxhighlight>

Now, run the following command and choose GCC 4.6 as alternative:

<syntaxhighlight lang=Bash>sudo update-alternatives --config gcc</syntaxhighlight>

Whenever you want to switch back to GCC 4.7 or newer, re-run the previous command. Please mind, that both C and C++ compilers are changed, not just C.

==Unsupported GPU architecture==

When compiling PCL with CUDA support enabled, you may get a <span style="color:#FF1493">''"nvcc fatal : Unsupported gpu architecture 'compute_52'"''</span> error or similar. This means that your current graphic card does not support some of the capabilities of the most modern versions of CUDA, and the SDK can not compile the binaries. You can see the compatibility info for your card model [https://en.wikipedia.org/wiki/CUDA#Supported_GPUs here].

In order to solve this, you must clean your PCL build folder by erasing all content:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

Configure again with CMake:

<syntaxhighlight lang=Bash>cmake ..
ccmake .</syntaxhighlight>

And look for the <span style="color:#FF1493">"CUDA_ARCH_BIN"</span> option. Edit it and remove any architectures that are not compatible with your GPU, like for example 5.2 (in the above case). Then, configure and generate as usual with <span style="color:#228B22">'''C'''</span> and <span style="color:#228B22">'''G'''</span>, and compile.

=Apt=

==Unmet dependencies==

If you get an error like <span style="color:#FF1493">''"Depends: X but it is not going to be installed"''</span> while installing a package through apt (''X'' being the name of another package), it probably means that there is a conflict between two repositories offering different versions of the same package (e.g. PCL and ROS ones). To solve this, you will have to completely remove one of the repositories and stick with the other.

Another possibility is that the package in question is just broken, maybe because it is deprecated and pending removal, or because you are using a very recent release of Ubuntu and there is no version for it. I used to get this error a lot when trying to install OpenNI, but I solved it by following the advice above. Also, you can try compiling everything from source.

==Trying to overwrite X, which is also in package Y==

During the installation of the PrimeSense drivers (using the ROS approach), you may get an error that says <span style="color:#FF1493">''"trying to overwrite '/etc/openni/GlobalDefaults.ini', which is also in package libopenni-sensor-pointclouds0"''</span>. This is actually a common problem, that can happen with other packages. It means that a file included in one of the new packages is also provided by another package that is already installed. In this case, it is the <span style="color:#1E90FF">''/etc/openni/GlobalDefaults.ini''</span> file, which is already included in the package <span style="color:#228B22">''libopenni-sensor-pointclouds0''</span>, making the installation of <span style="color:#228B22">''libopenni-sensor-primesense0''</span> fail.

In order to fix it, run the following command to manually retry the installation (change the file name as necessary):

<syntaxhighlight lang=Bash>sudo dpkg -i --force-overwrite /var/cache/apt/archives/libopenni-sensor-primesense0_5.1.0.41-3+trusty1_amd64.deb</syntaxhighlight>

This time, you will be prompted whether or not you want to overwrite the file. Choose yes, and then use the following command to make sure there are no packages left pending:

<syntaxhighlight lang=Bash>sudo apt-get -f install</syntaxhighlight>

=CMake=

==VTK warnings==

When running CMake to configure your PCL project before building it, you may get a lot of messages that say something like this:

<syntaxhighlight lang=make>-- The imported target "vtkPythonCore" references the file
"/usr/lib/libvtkPythonCore.so.5.8.0"
but this file does not exist. Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
"/usr/lib/vtk-5.8/VTKTargets-release.cmake"
but not all the files it references.</syntaxhighlight>

The program will build anyway, so this is not a big issue. But if you find the messages annoying, you can fix it by installing two packages:

<syntaxhighlight lang=Bash>sudo apt-get install python-vtk libvtk-java</syntaxhighlight>

==QVTK_LIBRARY_DEBUG or QVTK_LIBRARY_RELEASE not found==

If this error comes up when trying to configure the CMake build of PCL trunk, then you must install these two packages:

<syntaxhighlight lang=Bash>sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

Doing this will replace VTK 5 with version 6 and Qt 4 with version 5, which are the ones now officially supported by PCL 1.8.

=GCC=

==No such instruction==

When building PCL, it is possible that compilation will fail with a message like <span style="color:#FF1493">''"no such instruction: `vfmadd312ss 32(%rsp),%xmm15,%xmm7'"''</span>. What this means is that you have a combination of a brand new i7 CPU and an outdated compiler (such as GCC 4.6, which comes with Ubuntu 12.04). You could try to update to GCC 4.8, with it is a bit complicated, but there is an easier solution. Configure again the project with:

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

Now, you must press <span style="color:#228B22">'''T'''</span> to toggle advanced mode. Then, look for the <span style="color:#FF1493">"CMAKE_C_FLAGS"</span> and <span style="color:#FF1493">"CMAKE_CXX_FLAGS"</span> options. Edit them and add <span style="color:#FF1493">"-march=corei7"</span> (by default, <span style="color:#FF1493">"-march=native"</span> is set in the Makefiles). Configure and generate (<span style="color:#228B22">'''C'''</span> and <span style="color:#228B22">'''G'''</span>), and resume the compilation.

=PCL=

==Point type not properly registered==

You will get this error if you try to save or load a descriptor that does not have its own defined custom PointType (and uses <span style="color:#FF1493">"pcl::Histogram<>"</span>). See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|this]] for the solution.

=OpenNI=

==No device found / No devices connected==

===Microsoft Kinect===

If your system does not seem to be able to use Kinect, despite it being correctly plugged and listed by <span style="color:#228B22">''lsusb''</span>, then you may have a problem with the installed drivers. I eventually discovered that, in 32-bit Ubuntu, it was necessary to downgrade <span style="color:#228B22">''openni-dev''</span> and <span style="color:#228B22">''ps-engine''</span> to versions 1.3.2.1 and 5.0.3.3 respectively, using your package manager.

If you use <span style="color:#228B22">''Synaptic''</span>, the instructions are simple. Select both mentioned packages, press <span style="color:#228B22">'''Ctrl+E'''</span> to choose the desired version you want installed, and then apply the changes, accepting to overwrite any files. Then, to prevent them from being restored in the next upgrade, select each and click the <span style="color:#228B22">''"Lock Version"''</span> option in the <span style="color:#228B22">''"Package"''</span> menu. The packages will then be picked up by the <span style="color:#FF1493">"Status: Pinned"</span> filter on the left. Reboot, and everything should be working now.

[[Image:Synaptic_Kinect_lock.png|thumb|center|700px|Downgraded and locked ''openni-dev'' and ''ps-engine'' in Synaptic package manager.]]

===ASUS Xtion PRO===

In order to get our Xtion device working with OpenNI, we have to modify a configuration file:

<syntaxhighlight lang=Bash>sudo nano /etc/openni/GlobalDefaults.ini</syntaxhighlight>

If you installed OpenNI manually instead of using the repositories, the file is located in a different path:

<syntaxhighlight lang=Bash>sudo nano /usr/etc/primesense/GlobalDefaults.ini</syntaxhighlight>

Find (<span style="color:#228B22">'''Ctrl+W'''</span>) the line that says <span style="color:#FF1493">";UsbInterface=2"</span> and uncomment it (remove the semicolon at the start). Save the file (<span style="color:#228B22">'''Ctrl+O'''</span>) and close (<span style="color:#228B22">'''Ctrl+X'''</span>). Reboot your computer, and you should be able to use your Xtion device with OpenNI/PCL. Whenever you want to switch back to a Kinect, you must comment the line, or else you will get the same <span style="color:#FF1493">''"No devices connected"''</span> or <span style="color:#FF1493">''"USB interface is not supported!"''</span> error.

If you still get this error, or you have another problem, like inconsistent framerate or freeze-ups, check that your Xtion is not plugged to an USB 3.0 port. Sadly, the firmware has a bug that prevents the device from working with 3.0 ports. Either switch to a 2.0 one (if your computer does not have one, you can use a [https://en.wikipedia.org/wiki/USB_hub hub]) or try applying [https://github.com/nh2/asus-xtion-fix this fix].

==glInit() not found / OpenGL isues==

If you get this error while compiling OpenNI, and you are sure you have installed all dependencies, then this is due to faulty Makefiles that do not link against GL libraries where they should.

Go to the <span style="color:#FF8C00">''Platform/Linux/Build/Samples/''</span> subdirectory and find the samples that are giving errors (look at the compiler output to find them). For each one, enter its subdirectory and open the <span style="color:#1E90FF">''Makefile''</span> you will find there. Go to the line that says:

<syntaxhighlight lang=Bash>USED_LIBS += glut</syntaxhighlight>

And change it to the following:

<syntaxhighlight lang=Bash>USED_LIBS += glut GL</syntaxhighlight>

The samples should now compile.

==Permission denied==

If you get this error related to some header (.h) in <span style="color:#FF8C00">''/usr/include/''</span>, it means that OpenNI's installer has given the wrong permissions when moving files there, and the compiler can not access them. Locate the faulty file and change its permissions to allow reading for everyone:

<syntaxhighlight lang=Bash>sudo chmod +r -R /usr/include/<file></syntaxhighlight>

For example, a very drastic (and not recommended) way of fixing this for OpenNI's headers would be:

<syntaxhighlight lang=Bash>sudo chmod 0777 -R /usr/include/ni/</syntaxhighlight>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI troubleshooting

2015-11-01T23:28:23Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

This page tries to cover all of the issues I ran into while trying to get my Kinect device ready for developing with PCL. If you have any questions regarding this subject or know of a different bug, feel free to send me an email describing it, and I will add it to this section. Just remember to try Google first!

=CUDA=

==Unsupported configuration==

When you run the CUDA installer on a fairly modern Linux, you may get a message saying that you are trying to install it on an unsupported configuration. What this means is that CUDA can not work with the newest versions of the GCC compiler. As of version 7, the highest compatible GCC is 4.8.2 for Ubuntu 14.04 LTS and 4.9.1 for 14.10 (12.04 LTS has been deprecated). Check the compatibility table [http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#system-requirements here].

In order to solve this, you must install an older version, like 4.6 for Ubuntu 12.04, and set it as the "official" alternative for compiling (change 4.7 for your current version):

<syntaxhighlight lang=Bash>sudo apt-get install gcc-4.6 g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7</syntaxhighlight>

Now, run the following command and choose GCC 4.6 as alternative:

<syntaxhighlight lang=Bash>sudo update-alternatives --config gcc</syntaxhighlight>

Whenever you want to switch back to GCC 4.7 or newer, re-run the previous command. Please mind, that both C and C++ compilers are changed, not just C.

==Unsupported GPU architecture==

When compiling PCL with CUDA support enabled, you may get a <span style="color:#FF1493">''"nvcc fatal : Unsupported gpu architecture 'compute_52'"''</span> error or similar. This means that your current graphic card does not support some of the capabilities of the most modern versions of CUDA, and the SDK can not compile the binaries. You can see the compatibility info for your card model [https://en.wikipedia.org/wiki/CUDA#Supported_GPUs here].

In order to solve this, you must clean your PCL build folder by erasing all content:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

Configure again with CMake:

<syntaxhighlight lang=Bash>cmake ..
ccmake .</syntaxhighlight>

And look for the <span style="color:#FF1493">"CUDA_ARCH_BIN"</span> option. Edit it and remove any architectures that are not compatible with your GPU, like for example 5.2 (in the above case). Then, configure and generate as usual with <span style="color:#228B22">'''C'''</span> and <span style="color:#228B22">'''G'''</span>, and compile.

=Apt=

==Unmet dependencies==

If you get an error like <span style="color:#FF1493">''"Depends: X but it is not going to be installed"''</span> while installing a package through apt (''X'' being the name of another package), it probably means that there is a conflict between two repositories offering different versions of the same package (e.g. PCL and ROS ones). To solve this, you will have to completely remove one of the repositories and stick with the other.

Another possibility is that the package in question is just broken, maybe because it is deprecated and pending removal, or because you are using a very recent release of Ubuntu and there is no version for it. I used to get this error a lot when trying to install OpenNI, but I solved it by following the advice above. Also, you can try compiling everything from source.

==Trying to overwrite X, which is also in package Y==

During the installation of the PrimeSense drivers (using the ROS approach), you may get an error that says <span style="color:#FF1493">''"trying to overwrite '/etc/openni/GlobalDefaults.ini', which is also in package libopenni-sensor-pointclouds0"''</span>. This is actually a common problem, that can happen with other packages. It means that a file included in one of the new packages is also provided by another package that is already installed. In this case, it is the <span style="color:#1E90FF">''/etc/openni/GlobalDefaults.ini''</span> file, which is already included in the package <span style="color:#228B22">''libopenni-sensor-pointclouds0''</span>, making the installation of <span style="color:#228B22">''libopenni-sensor-primesense0''</span> fail.

In order to fix it, run the following command to manually retry the installation (change the file name as necessary):

<syntaxhighlight lang=Bash>sudo dpkg -i --force-overwrite /var/cache/apt/archives/libopenni-sensor-primesense0_5.1.0.41-3+trusty1_amd64.deb</syntaxhighlight>

This time, you will be prompted whether or not you want to overwrite the file. Choose yes, and then use the following command to make sure there are no packages left pending:

<syntaxhighlight lang=Bash>sudo apt-get -f install</syntaxhighlight>

=CMake=

==VTK warnings==

When running CMake to configure your PCL project before building it, you may get a lot of messages that say something like this:

<syntaxhighlight lang=make>-- The imported target "vtkPythonCore" references the file
"/usr/lib/libvtkPythonCore.so.5.8.0"
but this file does not exist. Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
"/usr/lib/vtk-5.8/VTKTargets-release.cmake"
but not all the files it references.</syntaxhighlight>

The program will build anyway, so this is not a big issue. But if you find the messages annoying, you can fix it by installing two packages:

<syntaxhighlight lang=Bash>sudo apt-get install python-vtk libvtk-java</syntaxhighlight>

==QVTK_LIBRARY_DEBUG or QVTK_LIBRARY_RELEASE not found==

If this error comes up when trying to configure the CMake build of PCL trunk, then you must install these two packages:

<syntaxhighlight lang=Bash>sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

Doing this will replace VTK 5 with version 6 and Qt 4 with version 5, which are the ones now officially supported by PCL 1.8.

=GCC=

==No such instruction==

When building PCL, it is possible that compilation will fail with a message like <span style="color:#FF1493">''"no such instruction: `vfmadd312ss 32(%rsp),%xmm15,%xmm7'"''</span>. What this means is that you have a combination of a brand new i7 CPU and an outdated compiler (such as GCC 4.6, which comes with Ubuntu 12.04). You could try to update to GCC 4.8, with it is a bit complicated, but there is an easier solution. Configure again the project with:

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

Now, you must press <span style="color:#228B22">'''T'''</span> to toggle advanced mode. Then, look for the <span style="color:#FF1493">"CMAKE_C_FLAGS"</span> and <span style="color:#FF1493">"CMAKE_CXX_FLAGS"</span> options. Edit them and add <span style="color:#FF1493">"-march=corei7"</span> (by default, <span style="color:#FF1493">"-march=native"</span> is set in the Makefiles). Configure and generate (<span style="color:#228B22">'''C'''</span> and <span style="color:#228B22">'''G'''</span>), and resume the compilation.

=PCL=

==Point type not properly registered==

You will get this error if you try to save or load a descriptor that does not have its own defined custom PointType (and uses <span style="color:#FF1493">"pcl::Histogram<>"</span>). See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|this]] for the solution.

=OpenNI=

==No device found / No devices connected==

===Microsoft Kinect===

If your system does not seem to be able to use Kinect, despite it being correctly plugged and listed by <span style="color:#228B22">''lsusb''</span>, then you may have a problem with the installed drivers. I eventually discovered that, in 32-bit Ubuntu, it was necessary to downgrade <span style="color:#228B22">''openni-dev''</span> and <span style="color:#228B22">''ps-engine''</span> to versions 1.3.2.1 and 5.0.3.3 respectively, using your package manager.

If you use <span style="color:#228B22">''Synaptic''</span>, the instructions are simple. Select both mentioned packages, press <span style="color:#228B22">'''Ctrl+E'''</span> to choose the desired version you want installed, and then apply the changes, accepting to overwrite any files. Then, to prevent them from being restored in the next upgrade, select each and click the <span style="color:#228B22">''"Lock Version"''</span> option in the <span style="color:#228B22">''"Package"''</span> menu. The packages will then be picked up by the <span style="color:#FF1493">"Status: Pinned"</span> filter on the left. Reboot, and everything should be working now.

[[Image:Synaptic_Kinect_lock.png|thumb|center|700px|Downgraded and locked ''openni-dev'' and ''ps-engine'' in Synaptic package manager.]]

===ASUS Xtion PRO===

In order to get our Xtion device working with OpenNI, we have to modify a configuration file:

<syntaxhighlight lang=Bash>sudo nano /etc/openni/GlobalDefaults.ini</syntaxhighlight>

If you installed OpenNI manually instead of using the repositories, the file is located in a different path:

<syntaxhighlight lang=Bash>sudo nano /usr/etc/primesense/GlobalDefaults.ini</syntaxhighlight>

Find (<span style="color:#228B22">'''Ctrl+W'''</span>) the line that says <span style="color:#FF1493">";UsbInterface=2"</span> and uncomment it (remove the semicolon at the start). Save the file (<span style="color:#228B22">'''Ctrl+O'''</span>) and close (<span style="color:#228B22">'''Ctrl+X'''</span>). Reboot your computer, and you should be able to use your Xtion device with OpenNI/PCL. Whenever you want to switch back to a Kinect, you must comment the line, or else you will get the same <span style="color:#FF1493">''"No devices connected"''</span> or <span style="color:#FF1493">''"USB interface is not supported!"''</span> error.

If you still get this error, or you have another problem, like inconsistent framerate or freeze-ups, check that your Xtion is not plugged to an USB 3.0 port. Sadly, the firmware has a bug that prevents the device from working with 3.0 ports. Either switch to a 2.0 one (if your computer does not have one, you can use a [http://en.wikipedia.org/wiki/USB_hub hub]) or try applying [https://github.com/nh2/asus-xtion-fix this fix].

==glInit() not found / OpenGL isues==

If you get this error while compiling OpenNI, and you are sure you have installed all dependencies, then this is due to faulty Makefiles that do not link against GL libraries where they should.

Go to the <span style="color:#FF8C00">''Platform/Linux/Build/Samples/''</span> subdirectory and find the samples that are giving errors (look at the compiler output to find them). For each one, enter its subdirectory and open the <span style="color:#1E90FF">''Makefile''</span> you will find there. Go to the line that says:

<syntaxhighlight lang=Bash>USED_LIBS += glut</syntaxhighlight>

And change it to the following:

<syntaxhighlight lang=Bash>USED_LIBS += glut GL</syntaxhighlight>

The samples should now compile.

==Permission denied==

If you get this error related to some header (.h) in <span style="color:#FF8C00">''/usr/include/''</span>, it means that OpenNI's installer has given the wrong permissions when moving files there, and the compiler can not access them. Locate the faulty file and change its permissions to allow reading for everyone:

<syntaxhighlight lang=Bash>sudo chmod +r -R /usr/include/<file></syntaxhighlight>

For example, a very drastic (and not recommended) way of fixing this for OpenNI's headers would be:

<syntaxhighlight lang=Bash>sudo chmod 0777 -R /usr/include/ni/</syntaxhighlight>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 5: 3D object recognition (pipeline)

2015-11-01T23:26:02Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

In our [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)|previous article]] we saw how 3D descriptors could be used to identify points or clusters. But in order to have a working object recognition system, many more things are necessary. The sequence of steps that we have to implement to make such a system is known as the ''pipeline''. This final article will explain how to do it. The scope of what we will talk about is very wide and many has been written, so you should consider this a simple introductory tutorial, to build a basic knowledge so you can experiment further.

=Overview=

Ideally, a 3D object recognition system should be able to grab clouds from the device, preprocess them, compute descriptors, compare them with the ones stored in our object database, and output all matches with their position and orientation in the scene, in real time. Several components must then be implemented to perform these sequential steps, each one taking as input the output of the previous. This pipeline will be different depending on what type of descriptor we are using: local or global.

[[Image:3D_recognition_pipeline.png|thumb|center|800px|Example of local and global 3D recognition pipelines in PCL (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

First of all, we have to ''train'' the system. Training means, in this case, creating a database will all the objects we want to be able to recognize, and the descriptors for every associated pose. Only after that we can implement the recognition pipeline. Also, there are some postprocessing steps that are not mandatory to perform but will yield better results if done, like pose refinement and hypothesis verification.

In the next sections we will go over every step, with some PCL code snippets, as always.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://www.pointclouds.org/assets/uploads/cglibs13_recognition.pdf 3D Object Recognition and 6DOF Pose Estimation]
** [http://www.pointclouds.org/assets/icra2013/ensembles.pdf Ensemble Learning for Object Recognition and Pose Estimation]
** [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Recognition.pdf 3D Object Recognition in Clutter with the Point Cloud Library]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

=Training=

As stated, we require a database with all the objects we intend to recognize later (it is possible to build a recognition system that works with object categories and high level descriptions, for example to find all things that "look" like a chair in the scene, but that is an entirely different matter). The most common way to do this is to take dozens of snapshots of every object from different angles. This can be done with a device such as a pan-tilt unit (which consists of a mechanical platform that can be rotated on two axes in known, precise amounts), or with a common table and a printed checkerboard pattern on it, to be used as ground truth to get the camera position and orientation. Of course, you need a physical copy of the object first.

[[Image:Pan_tilt_unit.jpg|thumb|center|350px|Pan-tilt unit (image from http://pointclouds.org/).]]

Another possibility is to do it virtually, rendering 3D models of the objects (modelled beforehand with CAD software like [https://en.wikipedia.org/wiki/Blender_%28software%29 Blender]) and using the Z buffer, which contains the depth information, to simulate the input a depth sensor would give. The perks of doing this include: no segmentation step is necessary to get the object cluster, and the ground truth pose information will be 100% exact.

[[Image:Virtual_scanner_tesselated_sphere.png|thumb|center|600px|Visualization of the virtual scanning process (image taken from a presentation by [http://users.acin.tuwien.ac.at/mzillich/?site=0 Michael Zillich]).]]

Many 3D object [http://pointclouds.org/media/ datasets] that are available for download have been created with one of these methods. Whichever you choose, be sure to be consistent and take all snapshots uniformly, with the same sampling level (one every X degrees). When choosing this amount, try to make a compromise between the complexity of the database (size is usually not a problem, but the more snapshots you have, the longer it will take to finish the matching step) and the precision of pose estimation.

After this, the desired descriptor(s) must be computed for every snapshot of each object, and saved to disk. If global descriptors are being used, a [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH|Camera Roll Histogram]] (CRH) should be included in order to retrieve the full 6 DoF pose, as many descriptors are invariant to the camera roll angle, which would limit the pose estimation to 5 DoF. Finally, ground truth information about the camera position and orientation will make it possible to compare it against the result given back by the recognition system. Most datasets include such information in text files next to the corresponding snapshot, usually in the form of a 3x3 rotation matrix and a 3D translation vector.

If you are using local descriptors, it is possible to work with a database of full objects, instead of partial snapshots (this can not be done with global descriptors because they are computed from the whole cluster, and a full object is something we would never get in practice; plus, they also encode viewpoint information). To do this, you would have to capture the object from all angles and then use registration or other technique to stitch the clouds together in order to create a complete, continous model. Resampling should also be needed because overlapping areas would have higher point density. Also, as normals are oriented according to the viewpoint when they are computed, you would have to merge them properly so they all point outwards, instead of recomputing them after composing the object cluster. The perk of using full objects is that the matching step will be shorter, as the system will not have to check against the descriptors of multiple snapshots.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
</div>

==Getting a full model==

Like I explained above, getting a full .pcd model for our object with a sensor requires some careful setup, good segmentation, normal computation and possibly resampling. On the other hand, if we use a 3D CAD model, it becomes much easier. A program like [https://en.wikipedia.org/wiki/Blender_%28software%29 Blender] (free and open source) can be used to design a model of the object, after taking some measurements. Programs like [https://en.wikipedia.org/wiki/MeshLab MeshLab] can convert to and from many types of 3D formats. In this case, the one favored by PCL is binary [https://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] with normals (Blender can export to ASCII PLY, but I have sometimes encountered a processing error with those).

===Raytracing===

PCL offers a couple of command line tools to render a .pcd cloud file from a CAD model. The first one does this by raytracing. It uses a 3D sphere or icosahedron that is tesselated (divided in polygonal regions). Then, the virtual camera is placed in each of the vertices (or the polygons' centers) pointing to the origin, where the object model is. Multiple snapshots are rendered, and the final cloud is composed from this information.

You can invoke it like this:

<syntaxhighlight lang=Bash>pcl_mesh2pcd <input.ply> <output.pcd></syntaxhighlight>

Here you can see the result of converting a mesh exported from Blender, the program's mascot, [https://en.wikipedia.org/wiki/Blender_%28software%29#Suzanne Suzanne]:

<center><gallery widths=300px>
File:Pcl_mesh2pcd_example_original.png | Original 3D mesh in VTK format.
File:Pcl_mesh2pcd_example_result.png | Resulting point cloud.
</gallery></center>

The tool has some parameters available, you can check their usage with:

<syntaxhighlight lang=Bash>pcl_mesh2pcd -h</syntaxhighlight>

For example, one of the parameters controls the size of the voxel grid used to downsample the cloud after the raytracing operation. With this you can regulate the point density of the resulting .pcd file.

===Sampling===

The second tool we are going to see employs sampling, with a voxel grid. It gets similar results to the previous one. You can invoke it in an identical way:

<syntaxhighlight lang=Bash>pcl_mesh_sampling <input.ply> <output.pcd></syntaxhighlight>

It has the same parameters as the previous tool. Check the usage with:

<syntaxhighlight lang=Bash>pcl_mesh_sampling -h</syntaxhighlight>

==Getting partial views==

===Tool===

If you need to get a collection of partial snapshots of the model, PCL has also a tool for that. It is the virtual equivalent of using one of those rotating tables I told you about. It works just like <span style="color:#228B22">''pcl_mesh2pcd''</span>, but instead of composing all renders into a single file, it saves each one to its own.

<syntaxhighlight lang=Bash>pcl_virtual_scanner <input.ply></syntaxhighlight>

Inside a directory, 42 .pcd files will be generated. You can check the usage by invoking it with no parameters; there are a couple of interesting options to add noise to the clouds, in order to better simulate the input from sensors like Kinect. If you need something more flexible (like a custom amount of snapshots, as the program does not let you change the number), see the next section.

===Code===

You can customize the snapshot generation process by making direct use of the <span style="color:#FF1493">"RenderViewsTesselatedSphere"</span> class, which is similar to what the previous tool uses internally. The code is the following:

<syntaxhighlight lang=CPP>#include <pcl/io/vtk_lib_io.h>
#include <vtkPolyDataMapper.h>
#include <pcl/apps/render_views_tesselated_sphere.h>

int
main(int argc, char** argv)
{
// Load the PLY model from a file.
vtkSmartPointer<vtkPLYReader> reader = vtkSmartPointer<vtkPLYReader>::New();
reader->SetFileName(argv[1]);
reader->Update();

// VTK is not exactly straightforward...
vtkSmartPointer < vtkPolyDataMapper > mapper = vtkSmartPointer<vtkPolyDataMapper>::New();
mapper->SetInputConnection(reader->GetOutputPort());
mapper->Update();

vtkSmartPointer<vtkPolyData> object = mapper->GetInput();

// Virtual scanner object.
pcl::apps::RenderViewsTesselatedSphere render_views;
render_views.addModelFromPolyData(object);
// Pixel width of the rendering window, it directly affects the snapshot file size.
render_views.setResolution(150);
// Horizontal FoV of the virtual camera.
render_views.setViewAngle(57.0f);
// If true, the resulting clouds of the snapshots will be organized.
render_views.setGenOrganized(true);
// How much to subdivide the icosahedron. Increasing this will result in a lot more snapshots.
render_views.setTesselationLevel(1);
// If true, the camera will be placed at the vertices of the triangles. If false, at the centers.
// This will affect the number of snapshots produced (if true, less will be made).
// True: 42 for level 1, 162 for level 2, 642 for level 3...
// False: 80 for level 1, 320 for level 2, 1280 for level 3...
render_views.setUseVertices(true);
// If true, the entropies (the amount of occlusions) will be computed for each snapshot (optional).
render_views.setComputeEntropies(true);

render_views.generateViews();

// Object for storing the rendered views.
std::vector<pcl::PointCloud<pcl::PointXYZ>::Ptr> views;
// Object for storing the poses, as 4x4 transformation matrices.
std::vector<Eigen::Matrix4f, Eigen::aligned_allocator<Eigen::Matrix4f> > poses;
// Object for storing the entropies (optional).
std::vector<float> entropies;
render_views.getViews(views);
render_views.getPoses(poses);
render_views.getEntropies(entropies);
}</syntaxhighlight>

At the end of the program, the generated views (as point clouds) and the corresponding poses (transformation matrices) get saved to those vectors, so you can use them as you like (e.g., saving the snapshots to .pcd files and the poses to text files). There is no API documentation available, if you want to know more, check the <span style="color:#1E90FF">''render_views_tesselated_sphere.h''</span> file inside PCL's source folder.

If you get a segmentation fault, check if the .ply file is in ASCII format (you should be able to open and read it with a text editor). If it is, convert it to binary .ply using MeshLab. This usually solves it.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': 3D model, [Resolution], [View angle], [Camera constraints], [Organized], [Sphere size]. [Tesselation level], [Use vertices], [Compute entropies]
* '''Output''': Snapshot clouds, Snapshot poses, [Snapshot entropies]
* [http://robotica.unileon.es/~victorm/PCL_snapshot_creator.tar.gz Download]
</div>

=Local pipeline=

The following sections enumerate and explain the steps that compose the object recognition pipeline when using local descriptors.

==Keypoint extraction==

In this step, we have to decide which points from the current cloud will be considered keypoints (and have the local descriptor computed for them). The main characteristics of a good keypoint detector are:

* '''Repeatability''': there should be a good chance of the same points being chosen over several iterations, even when the scene is captured from a different angle.
* '''Distinctiveness''': the chosen keypoints should be highly characterizing and descriptive. It should be easy to describe and match them.

Because the second one depends on the local descriptor being used (and how the feature is computed), a different keypoint detection technique would have to be implemented for each. Another simple alternative is to perform downsampling on the cloud, and use all remaining points. This can be done easily with [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Downsampling|downsampling]]. Follow the link to find a code snippet that you can adapt to use in your system. You can also use the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF keypoint detector]].

===ISS===

Another keypoint detector available in PCL is the ISS (Intrinsic Shape Signatures) one. ISS is a local descriptor presented in 2009, which creates a view-independent signature of the local surface patch. An algorithm for choosing keypoints was also included to better fit the descriptor. The algorithm scans the surfaces and chooses only points with large variations in the [https://en.wikipedia.org/wiki/Principal_curvature principal direction] (the shape of the surface), which are ideal for keypoints.

You can use the associated PCL class this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/iss_3d.h>

// This function by Tommaso Cavallari and Federico Tombari, taken from the tutorial
// http://pointclouds.org/documentation/tutorials/correspondence_grouping.php
double
computeCloudResolution(const pcl::PointCloud<pcl::PointXYZ>::ConstPtr& cloud)
{
double resolution = 0.0;
int numberOfPoints = 0;
int nres;
std::vector<int> indices(2);
std::vector<float> squaredDistances(2);
pcl::search::KdTree<pcl::PointXYZ> tree;
tree.setInputCloud(cloud);

for (size_t i = 0; i < cloud->size(); ++i)
{
if (! pcl_isfinite((*cloud)[i].x))
continue;

// Considering the second neighbor since the first is the point itself.
nres = tree.nearestKSearch(i, 2, indices, squaredDistances);
if (nres == 2)
{
resolution += sqrt(squaredDistances[1]);
++numberOfPoints;
}
}
if (numberOfPoints != 0)
resolution /= numberOfPoints;

return resolution;
}

int
main(int argc, char** argv)
{
// Objects for storing the point cloud and the keypoints.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr keypoints(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// ISS keypoint detector object.
pcl::ISSKeypoint3D<pcl::PointXYZ, pcl::PointXYZ> detector;
detector.setInputCloud(cloud);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
detector.setSearchMethod(kdtree);
double resolution = computeCloudResolution(cloud);
// Set the radius of the spherical neighborhood used to compute the scatter matrix.
detector.setSalientRadius(6 * resolution);
// Set the radius for the application of the non maxima supression algorithm.
detector.setNonMaxRadius(4 * resolution);
// Set the minimum number of neighbors that has to be found while applying the non maxima suppression algorithm.
detector.setMinNeighbors(5);
// Set the upper bound on the ratio between the second and the first eigenvalue.
detector.setThreshold21(0.975);
// Set the upper bound on the ratio between the third and the second eigenvalue.
detector.setThreshold32(0.975);
// Set the number of prpcessing threads to use. 0 sets it to automatic.
detector.setNumberOfThreads(4);

detector.compute(*keypoints);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Salient radius, Non maxima radius, Minimum neighbors, Eigenvalues thresholds, [Number of threads]
* '''Output''': Keypoints
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=5457637 Intrinsic shape signatures: A shape descriptor for 3D object recognition] (requires IEEE Xplore subscription) (Yu Zhong, 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_i_s_s_keypoint3_d.html pcl::ISSKeypoint3D]
* [http://robotica.unileon.es/~victorm/PCL_ISS_keypoints.tar.gz Download]
</div>

==Computing descriptors==

The next step is to compute the desired local descriptor(s) for every keypoint. The input parameters vary from one to another, and their efectiveness depends on the scene we are capturing and the objects we are working with, so I can not give you an ideal solution. Check the list of [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Table|available descriptors]], choose the one you deem best, and study the original publication to understand how it works and how the computation can be tuned to your needs, with the use of parameters (and try to understand how changing them affects the output). Refer to the given code snippets for each.

==Matching==

After the local descriptor has been computed for every keypoint in the cloud, we have to match them, finding correspondences with the ones we have stored in our object database. For this, a search structure like a [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#k-d_tree|''k''-d tree]] can be used to perform a nearest neighbor search, retrieving the Euclidean distances between descriptors (and optionally, enforcing a maximum distance value as a threshold). Every descriptor in the scene should be matched against the descriptors of every model in the database, because this accounts for the presence of multiple instances of the model, which would not be recognized if we did it the other way around.

The following code sample shows how to find all point correspondences between the cloud and a model:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the SHOT descriptors for the scene.
pcl::PointCloud<pcl::SHOT352>::Ptr sceneDescriptors(new pcl::PointCloud<pcl::SHOT352>());
// Object for storing the SHOT descriptors for the model.
pcl::PointCloud<pcl::SHOT352>::Ptr modelDescriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read the already computed descriptors from disk.
if (pcl::io::loadPCDFile<pcl::SHOT352>(argv[1], *sceneDescriptors) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::SHOT352>(argv[2], *modelDescriptors) != 0)
{
return -1;
}

// A kd-tree object that uses the FLANN library for fast search of nearest neighbors.
pcl::KdTreeFLANN<pcl::SHOT352> matching;
matching.setInputCloud(modelDescriptors);
// A Correspondence object stores the indices of the query and the match,
// and the distance/weight.
pcl::CorrespondencesPtr correspondences(new pcl::Correspondences());

// Check every descriptor computed for the scene.
for (size_t i = 0; i < sceneDescriptors->size(); ++i)
{
std::vector<int> neighbors(1);
std::vector<float> squaredDistances(1);
// Ignore NaNs.
if (pcl_isfinite(sceneDescriptors->at(i).descriptor[0]))
{
// Find the nearest neighbor (in descriptor space)...
int neighborCount = matching.nearestKSearch(sceneDescriptors->at(i), 1, neighbors, squaredDistances);
// ...and add a new correspondence if the distance is less than a threshold
// (SHOT distances are between 0 and 1, other descriptors use different metrics).
if (neighborCount == 1 && squaredDistances[0] < 0.25f)
{
pcl::Correspondence correspondence(neighbors[0], static_cast<int>(i), squaredDistances[0]);
correspondences->push_back(correspondence);
}
}
}
std::cout << "Found " << correspondences->size() << " correspondences." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_correspondence.html pcl::Correspondence]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_matching.tar.gz Download]
</div>

==Correspondence grouping==

Right now, all we have is a list of correspondences between keypoints in the scene and keypoints from some object(s) in the database. This does not necessarily mean that a given object is present in the scene. Consider this example: a box with two keypoints located on opposite corners. The distance between the points is known. The matching stage has found two keypoints in the scene with very similar descriptors, but as it turns out, the distance between them is way different. Unless we are taking non-rigid transformations into account (such as scaling), then we have to discard the idea that those points belong to the object.

This check can be implemented with an additional step called ''correspondence grouping''. Like the name states, it groups correspondences that are geometrically consistent (for the given object model) in clusters, and discards the ones that do not. Rotations and translations are permitted, but anything other than that will not meet the criteria. Also, as a minimum of 3 correspondences are required for retrieving the 6 DoF pose, groups with less than that can be ignored.

PCL offers a couple of classes to perform correspondence grouping.

===Geometric consistency===

This is the simplest method. It just iterates over all feature correspondences not yet grouped, and adds them to the current subset if they are geometrically consistent. The set resolution can be tuned to make the algorithm more or less forgiving when enforcing the consistency. For every subset (which should correspond with an instance of the model), this PCL class also computes the transformation (rotation and translation), making the next step ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_(pipeline)#Pose_estimation|pose estimation]]) unnecessary.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/correspondence.h>
#include <pcl/recognition/cg/geometric_consistency.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the keypoints of the scene and the model.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the unclustered and clustered correspondences.
pcl::CorrespondencesPtr correspondences(new pcl::Correspondences());
std::vector<pcl::Correspondences> clusteredCorrespondences;
// Object for storing the transformations (rotation plus translation).
std::vector<Eigen::Matrix4f, Eigen::aligned_allocator<Eigen::Matrix4f> > transformations;

// Read the keypoints from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sceneKeypoints) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *modelKeypoints) != 0)
{
return -1;
}

// Note: here you would compute the correspondences.
// It has been omitted here for simplicity.

// Object for correspondence grouping.
pcl::GeometricConsistencyGrouping<pcl::PointXYZ, pcl::PointXYZ> grouping;
grouping.setSceneCloud(sceneKeypoints);
grouping.setInputCloud(modelKeypoints);
grouping.setModelSceneCorrespondences(correspondences);
// Minimum cluster size. Default is 3 (as at least 3 correspondences
// are needed to compute the 6 DoF pose).
grouping.setGCThreshold(3);
// Resolution of the consensus set used to cluster correspondences together,
// in metric units. Default is 1.0.
grouping.setGCSize(0.01);

grouping.recognize(transformations, clusteredCorrespondences);

std::cout << "Model instances found: " << transformations.size() << std::endl << std::endl;
for (size_t i = 0; i < transformations.size(); i++)
{
std::cout << "Instance " << (i + 1) << ":" << std::endl;
std::cout << "\tHas " << clusteredCorrespondences[i].size() << " correspondences." << std::endl << std::endl;

Eigen::Matrix3f rotation = transformations[i].block<3, 3>(0, 0);
Eigen::Vector3f translation = transformations[i].block<3, 1>(0, 3);
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(0, 0), rotation(0, 1), rotation(0, 2));
printf("\t\tR = | %6.3f %6.3f %6.3f | \n", rotation(1, 0), rotation(1, 1), rotation(1, 2));
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(2, 0), rotation(2, 1), rotation(2, 2));
std::cout << std::endl;
printf("\t\tt = < %0.3f, %0.3f, %0.3f >\n", translation(0), translation(1), translation(2));
}
}</syntaxhighlight>

If you do not need to compute the transformations (or prefer to do it later in its own step), you can use the <span style="color:#FF1493">"cluster()"</span> function instead of the <span style="color:#FF1493">"recognize()"</span> one, which only takes one parameter (the object for the clustered correspondences).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (model), Points (scene), Correspondences, Minimum cluster size, Consensus set resolution
* '''Output''': Clustered correspondences, [Transformation]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_geometric_consistency_grouping.html pcl::GeometricConsistencyGrouping]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_GC_grouping.tar.gz Download]
</div>

===Hough voting===

This method is a bit more complex. It uses a technique known as [https://en.wikipedia.org/wiki/Hough_transform Hough transform], which was originally devised to perform line detection on 2D images, though it was later expanded to work with arbitrary shapes or higher dimensions. The method works with a voting procedure, with votes being cast on candidates that are found to be better suitable. If enough votes are accumulated for a given position in the space, then the shape is considered "found" there and its parameters are retrieved.

The voting procedure is carried out in a parameter space, which would have 6 dimensions in case a full pose needed to be estimated (rotation plus translation). Because that would be computationally expensive, the method implemented in PCL uses only a 3D Hough space, and employs local Reference Frames (RFs) to account for the remaining 3 DoF. Its computation can then be divided in two steps.

[[Image:Hough_voting_pipeline.png|thumb|center|600px|Use of the Hough voting scheme (red) in a 3D object recognition pipeline (image from [http://vision.deis.unibo.it/fede/papers/psivt10.pdf original paper]).]]

====Computing Reference Frames====

For every pair of correspondences, a local Reference Frame must be computed to retrieve the pose later. PCL offers a class that implements the BOrder Aware Repeatable Directions (BOARD) algorithm.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/board.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneCloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelCloud(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr sceneNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::Normal>::Ptr modelNormals(new pcl::PointCloud<pcl::Normal>);
// Objects for storing the keypoints.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the Reference Frames.
pcl::PointCloud<pcl::ReferenceFrame>::Ptr sceneRF(new pcl::PointCloud<pcl::ReferenceFrame>);
pcl::PointCloud<pcl::ReferenceFrame>::Ptr modelRF(new pcl::PointCloud<pcl::ReferenceFrame>);

// Read the scene and model clouds from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sceneCloud) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *modelCloud) != 0)
{
return -1;
}

// Note: here you would estimate the normals for the whole cloud, and choose
// the keypoints. It has been omitted here for simplicity.

// BOARD RF estimation object.
pcl::BOARDLocalReferenceFrameEstimation<pcl::PointXYZ, pcl::Normal, pcl::ReferenceFrame> rf;
// Search radius (maximum distance of the points used to estimate the X and Y axes
// of the BOARD Reference Frame for a given point).
rf.setRadiusSearch(0.02f);
// Check if support is complete, or has missing regions because it is too close to mesh borders.
rf.setFindHoles(true);

rf.setInputCloud(sceneKeypoints);
rf.setInputNormals(sceneNormals);
rf.setSearchSurface(sceneCloud);
rf.compute(*sceneRF);

rf.setInputCloud(modelKeypoints);
rf.setInputNormals(modelNormals);
rf.setSearchSurface(modelCloud);
rf.compute(*modelRF);
}</syntaxhighlight>

Be sure to check the API reference because the BOARD estimation object has many more parameters to tune.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search radius, [Tangent radius], [Margin array size], [Margin threshold], [Find holes], [Hole size threshold], [Steepness threshold]
* '''Output''': Reference Frames
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''Publication''':
** [http://vision.deis.unibo.it/LRF/LRF_repeatability_ICCV2011.pdf On the repeatability of the local reference frame for partial shape matching] (Alioscia Petrelli and Luigi Di Stefano, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_b_o_a_r_d_local_reference_frame_estimation.html pcl::BOARDLocalReferenceFrameEstimation]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_Hough_RFs.tar.gz Download]
</div>

====Clustering====

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/recognition/cg/hough_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the keypoints of the scene and the model.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the Reference Frames.
pcl::PointCloud<pcl::ReferenceFrame>::Ptr sceneRF(new pcl::PointCloud<pcl::ReferenceFrame>);
pcl::PointCloud<pcl::ReferenceFrame>::Ptr modelRF(new pcl::PointCloud<pcl::ReferenceFrame>);
// Objects for storing the unclustered and clustered correspondences.
pcl::CorrespondencesPtr correspondences(new pcl::Correspondences());
std::vector<pcl::Correspondences> clusteredCorrespondences;
// Object for storing the transformations (rotation plus translation).
std::vector<Eigen::Matrix4f, Eigen::aligned_allocator<Eigen::Matrix4f> > transformations;

// Read the keypoints from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sceneKeypoints) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *modelKeypoints) != 0)
{
return -1;
}

// Note: here you would compute the correspondences and the Reference Frames.
// It has been omitted here for simplicity.

// Object for correspondence grouping.
pcl::Hough3DGrouping<pcl::PointXYZ, pcl::PointXYZ, pcl::ReferenceFrame, pcl::ReferenceFrame> grouping;
grouping.setInputCloud(modelKeypoints);
grouping.setInputRf(modelRF);
grouping.setSceneCloud(sceneKeypoints);
grouping.setSceneRf(sceneRF);
grouping.setModelSceneCorrespondences(correspondences);
// Minimum cluster size. Default is 3 (as at least 3 correspondences
// are needed to compute the 6 DoF pose).
grouping.setHoughThreshold(3);
// Size of each bin in the Hough space.
grouping.setHoughBinSize(3);
// If true, the vote casting procedure will interpolate the score
// between neighboring bins in the Hough space.
grouping.setUseInterpolation(true);
// If true, the vote casting procedure will use the correspondence's
// weighted distance to compute the Hough voting score.
grouping.setUseDistanceWeight(false);

grouping.recognize(transformations, clusteredCorrespondences);

std::cout << "Model instances found: " << transformations.size() << std::endl << std::endl;
for (size_t i = 0; i < transformations.size(); i++)
{
std::cout << "Instance " << (i + 1) << ":" << std::endl;
std::cout << "\tHas " << clusteredCorrespondences[i].size() << " correspondences." << std::endl << std::endl;

Eigen::Matrix3f rotation = transformations[i].block<3, 3>(0, 0);
Eigen::Vector3f translation = transformations[i].block<3, 1>(0, 3);
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(0, 0), rotation(0, 1), rotation(0, 2));
printf("\t\tR = | %6.3f %6.3f %6.3f | \n", rotation(1, 0), rotation(1, 1), rotation(1, 2));
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(2, 0), rotation(2, 1), rotation(2, 2));
std::cout << std::endl;
printf("\t\tt = < %0.3f, %0.3f, %0.3f >\n", translation(0), translation(1), translation(2));
}
}</syntaxhighlight>

If you do not give the Reference Frames to the Hough grouping object, it will calculate them itself, but then you need to specify additional parameters. See the API for more details. Also, you can <span style="color:#FF1493">"cluster()"</span> instead of <span style="color:#FF1493">"recognize()"</span> if you want to skip the pose estimation now.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (model), Points (scene), Correspondences, Reference Frames (model), Reference Frames (scene), Hough bin size, Hough threshold, [Use distance weight], [Use interpolation]
* '''Output''': Clustered correspondences, [Transformation]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/psivt10.pdf Object recognition in 3D scenes with occlusions and clutter by Hough voting] (Federico Tombari and Luigi Di Stefano, 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_hough3_d_grouping.html pcl::Hough3DGrouping]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_Hough_grouping.tar.gz Download]
</div>

==Pose estimation==

The two correspondence grouping classes included in PCL already compute the full pose, but you can do it manually if you want, or if you want to refine the results of the previous step (removing correspondences that are not compatible with the same 6 DoF pose). You can use a method like RANSAC (RANdom SAmple Consensus), to get the rotation and translation of the rigid transformation that best fits the correspondences of a model instance. In a previous tutorial we mentioned that there was a technique called [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration|feature-based registration]] which was an alternative to ICP, and this is it.

Of course, PCL has some classes for this. The one we are going to see adds a prerejection step to the RANSAC loop, in order to discard hypotheses that are probably wrong. To do this, the algorithm forms polygons with the points of the input sets, and then checks pose-invariant geometrical constraints. To be precise, it makes sure that the polygon edges are of similar length. You can let it run for a specified number of iterations (as, like you already know, RANSAC never actually converges), or set a threshold.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/shot.h>
#include <pcl/registration/sample_consensus_prerejective.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the scene and the model.
pcl::PointCloud<pcl::PointXYZ>::Ptr scene(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr model(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr alignedModel(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the SHOT descriptors for the scene and model.
pcl::PointCloud<pcl::SHOT352>::Ptr sceneDescriptors(new pcl::PointCloud<pcl::SHOT352>());
pcl::PointCloud<pcl::SHOT352>::Ptr modelDescriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read the clouds from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *scene) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *model) != 0)
{
return -1;
}

// Note: here you would compute or load the descriptors for both
// the scene and the model. It has been omitted here for simplicity.

// Object for pose estimation.
pcl::SampleConsensusPrerejective<pcl::PointXYZ, pcl::PointXYZ, pcl::SHOT352> pose;
pose.setInputSource(model);
pose.setInputTarget(scene);
pose.setSourceFeatures(modelDescriptors);
pose.setTargetFeatures(sceneDescriptors);
// Instead of matching a descriptor with its nearest neighbor, choose randomly between
// the N closest ones, making it more robust to outliers, but increasing time.
pose.setCorrespondenceRandomness(2);
// Set the fraction (0-1) of inlier points required for accepting a transformation.
// At least this number of points will need to be aligned to accept a pose.
pose.setInlierFraction(0.25f);
// Set the number of samples to use during each iteration (minimum for 6 DoF is 3).
pose.setNumberOfSamples(3);
// Set the similarity threshold (0-1) between edge lengths of the polygons. The
// closer to 1, the more strict the rejector will be, probably discarding acceptable poses.
pose.setSimilarityThreshold(0.6f);
// Set the maximum distance threshold between two correspondent points in source and target.
// If the distance is larger, the points will be ignored in the alignment process.
pose.setMaxCorrespondenceDistance(0.01f);

pose.align(*alignedModel);

if (pose.hasConverged())
{
Eigen::Matrix4f transformation = pose.getFinalTransformation();
Eigen::Matrix3f rotation = transformation.block<3, 3>(0, 0);
Eigen::Vector3f translation = transformation.block<3, 1>(0, 3);

std::cout << "Transformation matrix:" << std::endl << std::endl;
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(0, 0), rotation(0, 1), rotation(0, 2));
printf("\t\tR = | %6.3f %6.3f %6.3f | \n", rotation(1, 0), rotation(1, 1), rotation(1, 2));
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(2, 0), rotation(2, 1), rotation(2, 2));
std::cout << std::endl;
printf("\t\tt = < %0.3f, %0.3f, %0.3f >\n", translation(0), translation(1), translation(2));
}
else std::cout << "Did not converge." << std::endl;
}</syntaxhighlight>

For an alternative, check the [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_registration.html pcl::SampleConsensusModelRegistration] or the [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_initial_alignment.html pcl::SampleConsensusInitialAlignment] classes.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (model), Points (scene), Descriptors (model), Descriptors (scene), [Correspondence randomness], [Inlier fraction], [Number of samples], [Similarity threshold]
* '''Output''': Aligned model, [Transformation]
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/alignment_prerejective.php Robust pose estimation of rigid objects]
** [http://pointclouds.org/documentation/tutorials/template_alignment.php Aligning object templates to a point cloud]
* '''Publication''':
** [http://personal.lut.fi/users/joni.kamarainen/downloads/publications/icra2013.pdf Pose Estimation using Local Structure-Specific Shape and Appearance Context] (Anders Glent Buch et al., 2013)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_prerejective.html pcl::SampleConsensusPrerejective]
** [http://docs.pointclouds.org/trunk/classpcl_1_1registration_1_1_correspondence_rejector_poly.html pcl::registration::CorrespondenceRejectorPoly]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_pose.tar.gz Download]
</div>

=Global pipeline=

Because of the way global descriptors work, the global pipeline for object recognition differs in some steps.

==Segmentation==

Unlike local descriptors, global ones understand the notion of ''object''. They are not computed for single points, but for whole clusters. Because of this, the first step is to perform [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]] on the cloud in order to retrieve all objects. You can use any method you like from the ones available in PCL, as long as it works for you. Also, performing some kind of preprocessing is a good idea, like removing all clusters that are smaller or bigger than certain thresholds, which should be set according to the objects in the database. Yet another possibility is to perform [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Plane_model|plane segmentation]] to find any table(s) in the scene, and consider only clusters sitting on it (if we can assume there is a table, of course).

PCL provides the [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_polygonal_prism_data.html pcl::ExtractPolygonalPrismData] class for the latter task. By giving a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Convex_hull|convex hull]] computed from the plane coefficients, it will extrude it a certain height to create a prism, and then give back all points that lie inside. This is the code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/surface/convex_hull.h>
#include <pcl/segmentation/extract_polygonal_prism_data.h>
#include <pcl/visualization/cloud_viewer.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr plane(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr convexHull(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr objects(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Get the plane model, if present.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setOptimizeCoefficients(true);
pcl::PointIndices::Ptr planeIndices(new pcl::PointIndices);
segmentation.segment(*planeIndices, *coefficients);

if (planeIndices->indices.size() == 0)
std::cout << "Could not find a plane in the scene." << std::endl;
else
{
// Copy the points of the plane to a new cloud.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloud);
extract.setIndices(planeIndices);
extract.filter(*plane);

// Retrieve the convex hull.
pcl::ConvexHull<pcl::PointXYZ> hull;
hull.setInputCloud(plane);
// Make sure that the resulting hull is bidimensional.
hull.setDimension(2);
hull.reconstruct(*convexHull);

// Redundant check.
if (hull.getDimension() == 2)
{
// Prism object.
pcl::ExtractPolygonalPrismData<pcl::PointXYZ> prism;
prism.setInputCloud(cloud);
prism.setInputPlanarHull(convexHull);
// First parameter: minimum Z value. Set to 0, segments objects lying on the plane (can be negative).
// Second parameter: maximum Z value, set to 10cm. Tune it according to the height of the objects you expect.
prism.setHeightLimits(0.0f, 0.1f);
pcl::PointIndices::Ptr objectIndices(new pcl::PointIndices);

prism.segment(*objectIndices);

// Get and show all points retrieved by the hull.
extract.setIndices(objectIndices);
extract.filter(*objects);
pcl::visualization::CloudViewer viewerObjects("Objects on table");
viewerObjects.showCloud(objects);
while (!viewerObjects.wasStopped())
{
// Do nothing but wait.
}
}
else std::cout << "The chosen hull is not planar." << std::endl;
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:Prism_scene_before.png | Original scene, with a mug on a table.
File:Prism_scene_after.png | Result after using the polygonal prism class.
</gallery></center>

If you get some residual points of the table in the output, try increasing the minimum Z value a bit.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Convex hull, Height limits, [Dimensionality]
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_polygonal_prism_data.html pcl::ExtractPolygonalPrismData]
* [http://robotica.unileon.es/~victorm/PCL_polygonal_prism.tar.gz Download]
</div>

==Computing descriptors==

For every cluster that has survived the previous step, a global descriptor must be computed. Most only output one histogram (except for CVFH and derivatives), so the database will be smaller than with local descriptors. Check the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Table|list]] and choose the one that fits you best.

===CRH===

In a previous article about global descriptors we talked about how they were invariant to the camera roll angle. When [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#CVFH|CVFH]] was presented, the computation of a Camera Roll Histogram (CRH) was proposed to overcome this. The CRH should be computed and stored next to the global descriptor (VFH, CVFH...) for its use in a later phase, the pose estimation.

The CRH is computed as follows: for every point, its normal is projected onto a plane orthogonal to the vector given by the camera center and the centroid of the cluster (VFH) or region (CVFH). Then, the angle of the projected normal to the up vector of the camera is computed and added to the histogram, which has 90 bins (for a resolution of 4º). The projected normals are weighted by their magnitudes to reduce the effect of input noise. Check the [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 paper] for details.

You can compute it using PCL, of course:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/common/centroid.h>
#include <pcl/features/crh.h>

// A handy typedef.
typedef pcl::Histogram<90> CRH90;

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CRH.
pcl::PointCloud<CRH90>::Ptr histogram(new pcl::PointCloud<CRH90>);

// Note: this cloud file should contain a snapshot of the object. Remember
// that you need to compute a CRH for every VFH or CVFH descriptor that you
// are going to have (that is, one for every snapshot).

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CRH estimation object.
pcl::CRHEstimation<pcl::PointXYZ, pcl::Normal, CRH90> crh;
crh.setInputCloud(object);
crh.setInputNormals(normals);
Eigen::Vector4f centroid;
pcl::compute3DCentroid(*object, centroid);
crh.setCentroid(centroid);

// Compute the CRH.
crh.compute(*histogram);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Centroid
* '''Output''': Camera roll histogram
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_estimation.html pcl::CRHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CRH.tar.gz Download]
</div>

==Matching==

Like with the local pipeline, once all the descriptors have been computed we have to perform a search for their nearest neighbors in the database. The [https://en.wikipedia.org/wiki/Taxicab_geometry Manhattan or L1 distance] is recommended over the [https://en.wikipedia.org/wiki/Euclidean_distance Euclidean or L2] one because it is more robust to occlusions.

Usually, more than one neighbor is retrieved, which means that the found object's pose is somewhere between the two. The next steps can help improving the accuraccy of the estimation.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
</div>

==Pose estimation==

The object's position can be determined by computing and aligning the [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Computing_the_centroid|centroids]] of the clusters (the one in the scene, and the one in the snapshot). For the rotation, we must use the viewpoint information encoded in the descriptor (you can check the ground truth pose information that was saved with the matched snapshot). This still leaves the roll angle unresolved, as global descriptors are invariant to it. Now is when the Camera Roll Histogram ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH|CRH]]) that we should have stored next to the descriptor comes in handy. The angle can be obtained by aligning the current CRH with the stored one, completing the 6 DoF pose estimation.

===CRH===

This is the code for retrieving the roll angle using CRH:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>
#include <pcl/features/crh.h>
#include <pcl/recognition/crh_alignment.h>

#include <iostream>
#include <vector>

// A handy typedef.
typedef pcl::Histogram<90> CRH90;

int
main(int argc, char** argv)
{
// Clouds for storing the object's cluster and view.
pcl::PointCloud<pcl::PointXYZ>::Ptr viewCloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr clusterCloud(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the CRHs of both.
pcl::PointCloud<CRH90>::Ptr viewCRH(new pcl::PointCloud<CRH90>);
pcl::PointCloud<CRH90>::Ptr clusterCRH(new pcl::PointCloud<CRH90>);
// Objects for storing the centroids.
Eigen::Vector4f viewCentroid;
Eigen::Vector4f clusterCentroid;

// Read the trained view and the scene cluster from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *viewCloud) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *clusterCloud) != 0)
{
return -1;
}

// Note: here you would compute the CRHs and centroids of both clusters.
// It has been omitted here for simplicity.

// CRH alignment object.
pcl::CRHAlignment<pcl::PointXYZ, 90> alignment;
alignment.setInputAndTargetView(clusterCloud, viewCloud);
// CRHAlignment works with Vector3f, not Vector4f.
Eigen::Vector3f viewCentroid3f(viewCentroid[0], viewCentroid[1], viewCentroid[2]);
Eigen::Vector3f clusterCentroid3f(clusterCentroid[0], clusterCentroid[1], clusterCentroid[2]);
alignment.setInputAndTargetCentroids(clusterCentroid3f, viewCentroid3f);

// Compute the roll angle(s).
std::vector<float> angles;
alignment.computeRollAngle(*clusterCRH, *viewCRH, angles);

if (angles.size() > 0)
{
std::cout << "List of angles where the histograms correlate:" << std::endl;

for (int i = 0; i < angles.size(); i++)
{
std::cout << "\t" << angles.at(i) << " degrees." << std::endl;
}
}
}</syntaxhighlight>

This will return a list of most probable roll angles. If you want, you can instead use the [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_alignment.html#a32c40356d56e22d48eccce5e27e436f2 align()] function, which calls <span style="color:#FF1493">"computeRollAngle()"</span> internally. It will return a list of transformations that include the translation (computed from the difference between the centroid's coordinates) and the roll angles, but not the yaw or pitch (you have to get those with pose estimation).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (input and target), Centroids (input and target), CRHs (input and target)
* '''Output''': Roll angles, [Transformations]
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_alignment.html pcl::CRHAlignment]
* [http://robotica.unileon.es/~victorm/PCL_roll_estimation.tar.gz Download]
</div>

=Postprocessing=

The pipelines, as they are, should already return a decent result, but we can implement optional steps to improve the accuraccy of the pose estimation, and decrease the probability of a false positive.

==Pose refinement==

The 6 DoF pose estimation can be refined with the use of the [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Iterative_Closest_Point_.28ICP.29|Iterative Closest Point]] (ICP) algorithm. It will try to continuously realign the clouds to improve the transformation, until the termination condition is met (maximum iterations, and distance or error threshold). This is identical to performing [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Registration|registration]], only between input and model.

==Hypothesis verification==

''(Work in progress)''

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 5: 3D object recognition (pipeline)

2015-11-01T23:23:58Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

In our [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)|previous article]] we saw how 3D descriptors could be used to identify points or clusters. But in order to have a working object recognition system, many more things are necessary. The sequence of steps that we have to implement to make such a system is known as the ''pipeline''. This final article will explain how to do it. The scope of what we will talk about is very wide and many has been written, so you should consider this a simple introductory tutorial, to build a basic knowledge so you can experiment further.

=Overview=

Ideally, a 3D object recognition system should be able to grab clouds from the device, preprocess them, compute descriptors, compare them with the ones stored in our object database, and output all matches with their position and orientation in the scene, in real time. Several components must then be implemented to perform these sequential steps, each one taking as input the output of the previous. This pipeline will be different depending on what type of descriptor we are using: local or global.

[[Image:3D_recognition_pipeline.png|thumb|center|800px|Example of local and global 3D recognition pipelines in PCL (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

First of all, we have to ''train'' the system. Training means, in this case, creating a database will all the objects we want to be able to recognize, and the descriptors for every associated pose. Only after that we can implement the recognition pipeline. Also, there are some postprocessing steps that are not mandatory to perform but will yield better results if done, like pose refinement and hypothesis verification.

In the next sections we will go over every step, with some PCL code snippets, as always.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://www.pointclouds.org/assets/uploads/cglibs13_recognition.pdf 3D Object Recognition and 6DOF Pose Estimation]
** [http://www.pointclouds.org/assets/icra2013/ensembles.pdf Ensemble Learning for Object Recognition and Pose Estimation]
** [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Recognition.pdf 3D Object Recognition in Clutter with the Point Cloud Library]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

=Training=

As stated, we require a database with all the objects we intend to recognize later (it is possible to build a recognition system that works with object categories and high level descriptions, for example to find all things that "look" like a chair in the scene, but that is an entirely different matter). The most common way to do this is to take dozens of snapshots of every object from different angles. This can be done with a device such as a pan-tilt unit (which consists of a mechanical platform that can be rotated on two axes in known, precise amounts), or with a common table and a printed checkerboard pattern on it, to be used as ground truth to get the camera position and orientation. Of course, you need a physical copy of the object first.

[[Image:Pan_tilt_unit.jpg|thumb|center|350px|Pan-tilt unit (image from http://pointclouds.org/).]]

Another possibility is to do it virtually, rendering 3D models of the objects (modelled beforehand with CAD software like [http://www.blender.org/ Blender]) and using the Z buffer, which contains the depth information, to simulate the input a depth sensor would give. The perks of doing this include: no segmentation step is necessary to get the object cluster, and the ground truth pose information will be 100% exact.

[[Image:Virtual_scanner_tesselated_sphere.png|thumb|center|600px|Visualization of the virtual scanning process (image taken from a presentation by [http://users.acin.tuwien.ac.at/mzillich/?site=0 Michael Zillich]).]]

Many 3D object [http://pointclouds.org/media/ datasets] that are available for download have been created with one of these methods. Whichever you choose, be sure to be consistent and take all snapshots uniformly, with the same sampling level (one every X degrees). When choosing this amount, try to make a compromise between the complexity of the database (size is usually not a problem, but the more snapshots you have, the longer it will take to finish the matching step) and the precision of pose estimation.

After this, the desired descriptor(s) must be computed for every snapshot of each object, and saved to disk. If global descriptors are being used, a [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH|Camera Roll Histogram]] (CRH) should be included in order to retrieve the full 6 DoF pose, as many descriptors are invariant to the camera roll angle, which would limit the pose estimation to 5 DoF. Finally, ground truth information about the camera position and orientation will make it possible to compare it against the result given back by the recognition system. Most datasets include such information in text files next to the corresponding snapshot, usually in the form of a 3x3 rotation matrix and a 3D translation vector.

If you are using local descriptors, it is possible to work with a database of full objects, instead of partial snapshots (this can not be done with global descriptors because they are computed from the whole cluster, and a full object is something we would never get in practice; plus, they also encode viewpoint information). To do this, you would have to capture the object from all angles and then use registration or other technique to stitch the clouds together in order to create a complete, continous model. Resampling should also be needed because overlapping areas would have higher point density. Also, as normals are oriented according to the viewpoint when they are computed, you would have to merge them properly so they all point outwards, instead of recomputing them after composing the object cluster. The perk of using full objects is that the matching step will be shorter, as the system will not have to check against the descriptors of multiple snapshots.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
</div>

==Getting a full model==

Like I explained above, getting a full .pcd model for our object with a sensor requires some careful setup, good segmentation, normal computation and possibly resampling. On the other hand, if we use a 3D CAD model, it becomes much easier. A program like [http://www.blender.org/ Blender] (free and open source) can be used to design a model of the object, after taking some measurements. Programs like [http://meshlab.sourceforge.net/ MeshLab] can convert to and from many types of 3D formats. In this case, the one favored by PCL is binary [https://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] with normals (Blender can export to ASCII PLY, but I have sometimes encountered a processing error with those).

===Raytracing===

PCL offers a couple of command line tools to render a .pcd cloud file from a CAD model. The first one does this by raytracing. It uses a 3D sphere or icosahedron that is tesselated (divided in polygonal regions). Then, the virtual camera is placed in each of the vertices (or the polygons' centers) pointing to the origin, where the object model is. Multiple snapshots are rendered, and the final cloud is composed from this information.

You can invoke it like this:

<syntaxhighlight lang=Bash>pcl_mesh2pcd <input.ply> <output.pcd></syntaxhighlight>

Here you can see the result of converting a mesh exported from Blender, the program's mascot, [https://en.wikipedia.org/wiki/Blender_%28software%29#Suzanne Suzanne]:

<center><gallery widths=300px>
File:Pcl_mesh2pcd_example_original.png | Original 3D mesh in VTK format.
File:Pcl_mesh2pcd_example_result.png | Resulting point cloud.
</gallery></center>

The tool has some parameters available, you can check their usage with:

<syntaxhighlight lang=Bash>pcl_mesh2pcd -h</syntaxhighlight>

For example, one of the parameters controls the size of the voxel grid used to downsample the cloud after the raytracing operation. With this you can regulate the point density of the resulting .pcd file.

===Sampling===

The second tool we are going to see employs sampling, with a voxel grid. It gets similar results to the previous one. You can invoke it in an identical way:

<syntaxhighlight lang=Bash>pcl_mesh_sampling <input.ply> <output.pcd></syntaxhighlight>

It has the same parameters as the previous tool. Check the usage with:

<syntaxhighlight lang=Bash>pcl_mesh_sampling -h</syntaxhighlight>

==Getting partial views==

===Tool===

If you need to get a collection of partial snapshots of the model, PCL has also a tool for that. It is the virtual equivalent of using one of those rotating tables I told you about. It works just like <span style="color:#228B22">''pcl_mesh2pcd''</span>, but instead of composing all renders into a single file, it saves each one to its own.

<syntaxhighlight lang=Bash>pcl_virtual_scanner <input.ply></syntaxhighlight>

Inside a directory, 42 .pcd files will be generated. You can check the usage by invoking it with no parameters; there are a couple of interesting options to add noise to the clouds, in order to better simulate the input from sensors like Kinect. If you need something more flexible (like a custom amount of snapshots, as the program does not let you change the number), see the next section.

===Code===

You can customize the snapshot generation process by making direct use of the <span style="color:#FF1493">"RenderViewsTesselatedSphere"</span> class, which is similar to what the previous tool uses internally. The code is the following:

<syntaxhighlight lang=CPP>#include <pcl/io/vtk_lib_io.h>
#include <vtkPolyDataMapper.h>
#include <pcl/apps/render_views_tesselated_sphere.h>

int
main(int argc, char** argv)
{
// Load the PLY model from a file.
vtkSmartPointer<vtkPLYReader> reader = vtkSmartPointer<vtkPLYReader>::New();
reader->SetFileName(argv[1]);
reader->Update();

// VTK is not exactly straightforward...
vtkSmartPointer < vtkPolyDataMapper > mapper = vtkSmartPointer<vtkPolyDataMapper>::New();
mapper->SetInputConnection(reader->GetOutputPort());
mapper->Update();

vtkSmartPointer<vtkPolyData> object = mapper->GetInput();

// Virtual scanner object.
pcl::apps::RenderViewsTesselatedSphere render_views;
render_views.addModelFromPolyData(object);
// Pixel width of the rendering window, it directly affects the snapshot file size.
render_views.setResolution(150);
// Horizontal FoV of the virtual camera.
render_views.setViewAngle(57.0f);
// If true, the resulting clouds of the snapshots will be organized.
render_views.setGenOrganized(true);
// How much to subdivide the icosahedron. Increasing this will result in a lot more snapshots.
render_views.setTesselationLevel(1);
// If true, the camera will be placed at the vertices of the triangles. If false, at the centers.
// This will affect the number of snapshots produced (if true, less will be made).
// True: 42 for level 1, 162 for level 2, 642 for level 3...
// False: 80 for level 1, 320 for level 2, 1280 for level 3...
render_views.setUseVertices(true);
// If true, the entropies (the amount of occlusions) will be computed for each snapshot (optional).
render_views.setComputeEntropies(true);

render_views.generateViews();

// Object for storing the rendered views.
std::vector<pcl::PointCloud<pcl::PointXYZ>::Ptr> views;
// Object for storing the poses, as 4x4 transformation matrices.
std::vector<Eigen::Matrix4f, Eigen::aligned_allocator<Eigen::Matrix4f> > poses;
// Object for storing the entropies (optional).
std::vector<float> entropies;
render_views.getViews(views);
render_views.getPoses(poses);
render_views.getEntropies(entropies);
}</syntaxhighlight>

At the end of the program, the generated views (as point clouds) and the corresponding poses (transformation matrices) get saved to those vectors, so you can use them as you like (e.g., saving the snapshots to .pcd files and the poses to text files). There is no API documentation available, if you want to know more, check the <span style="color:#1E90FF">''render_views_tesselated_sphere.h''</span> file inside PCL's source folder.

If you get a segmentation fault, check if the .ply file is in ASCII format (you should be able to open and read it with a text editor). If it is, convert it to binary .ply using MeshLab. This usually solves it.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': 3D model, [Resolution], [View angle], [Camera constraints], [Organized], [Sphere size]. [Tesselation level], [Use vertices], [Compute entropies]
* '''Output''': Snapshot clouds, Snapshot poses, [Snapshot entropies]
* [http://robotica.unileon.es/~victorm/PCL_snapshot_creator.tar.gz Download]
</div>

=Local pipeline=

The following sections enumerate and explain the steps that compose the object recognition pipeline when using local descriptors.

==Keypoint extraction==

In this step, we have to decide which points from the current cloud will be considered keypoints (and have the local descriptor computed for them). The main characteristics of a good keypoint detector are:

* '''Repeatability''': there should be a good chance of the same points being chosen over several iterations, even when the scene is captured from a different angle.
* '''Distinctiveness''': the chosen keypoints should be highly characterizing and descriptive. It should be easy to describe and match them.

Because the second one depends on the local descriptor being used (and how the feature is computed), a different keypoint detection technique would have to be implemented for each. Another simple alternative is to perform downsampling on the cloud, and use all remaining points. This can be done easily with [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Downsampling|downsampling]]. Follow the link to find a code snippet that you can adapt to use in your system. You can also use the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF keypoint detector]].

===ISS===

Another keypoint detector available in PCL is the ISS (Intrinsic Shape Signatures) one. ISS is a local descriptor presented in 2009, which creates a view-independent signature of the local surface patch. An algorithm for choosing keypoints was also included to better fit the descriptor. The algorithm scans the surfaces and chooses only points with large variations in the [https://en.wikipedia.org/wiki/Principal_curvature principal direction] (the shape of the surface), which are ideal for keypoints.

You can use the associated PCL class this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/iss_3d.h>

// This function by Tommaso Cavallari and Federico Tombari, taken from the tutorial
// http://pointclouds.org/documentation/tutorials/correspondence_grouping.php
double
computeCloudResolution(const pcl::PointCloud<pcl::PointXYZ>::ConstPtr& cloud)
{
double resolution = 0.0;
int numberOfPoints = 0;
int nres;
std::vector<int> indices(2);
std::vector<float> squaredDistances(2);
pcl::search::KdTree<pcl::PointXYZ> tree;
tree.setInputCloud(cloud);

for (size_t i = 0; i < cloud->size(); ++i)
{
if (! pcl_isfinite((*cloud)[i].x))
continue;

// Considering the second neighbor since the first is the point itself.
nres = tree.nearestKSearch(i, 2, indices, squaredDistances);
if (nres == 2)
{
resolution += sqrt(squaredDistances[1]);
++numberOfPoints;
}
}
if (numberOfPoints != 0)
resolution /= numberOfPoints;

return resolution;
}

int
main(int argc, char** argv)
{
// Objects for storing the point cloud and the keypoints.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr keypoints(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// ISS keypoint detector object.
pcl::ISSKeypoint3D<pcl::PointXYZ, pcl::PointXYZ> detector;
detector.setInputCloud(cloud);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
detector.setSearchMethod(kdtree);
double resolution = computeCloudResolution(cloud);
// Set the radius of the spherical neighborhood used to compute the scatter matrix.
detector.setSalientRadius(6 * resolution);
// Set the radius for the application of the non maxima supression algorithm.
detector.setNonMaxRadius(4 * resolution);
// Set the minimum number of neighbors that has to be found while applying the non maxima suppression algorithm.
detector.setMinNeighbors(5);
// Set the upper bound on the ratio between the second and the first eigenvalue.
detector.setThreshold21(0.975);
// Set the upper bound on the ratio between the third and the second eigenvalue.
detector.setThreshold32(0.975);
// Set the number of prpcessing threads to use. 0 sets it to automatic.
detector.setNumberOfThreads(4);

detector.compute(*keypoints);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Salient radius, Non maxima radius, Minimum neighbors, Eigenvalues thresholds, [Number of threads]
* '''Output''': Keypoints
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=5457637 Intrinsic shape signatures: A shape descriptor for 3D object recognition] (requires IEEE Xplore subscription) (Yu Zhong, 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_i_s_s_keypoint3_d.html pcl::ISSKeypoint3D]
* [http://robotica.unileon.es/~victorm/PCL_ISS_keypoints.tar.gz Download]
</div>

==Computing descriptors==

The next step is to compute the desired local descriptor(s) for every keypoint. The input parameters vary from one to another, and their efectiveness depends on the scene we are capturing and the objects we are working with, so I can not give you an ideal solution. Check the list of [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Table|available descriptors]], choose the one you deem best, and study the original publication to understand how it works and how the computation can be tuned to your needs, with the use of parameters (and try to understand how changing them affects the output). Refer to the given code snippets for each.

==Matching==

After the local descriptor has been computed for every keypoint in the cloud, we have to match them, finding correspondences with the ones we have stored in our object database. For this, a search structure like a [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#k-d_tree|''k''-d tree]] can be used to perform a nearest neighbor search, retrieving the Euclidean distances between descriptors (and optionally, enforcing a maximum distance value as a threshold). Every descriptor in the scene should be matched against the descriptors of every model in the database, because this accounts for the presence of multiple instances of the model, which would not be recognized if we did it the other way around.

The following code sample shows how to find all point correspondences between the cloud and a model:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the SHOT descriptors for the scene.
pcl::PointCloud<pcl::SHOT352>::Ptr sceneDescriptors(new pcl::PointCloud<pcl::SHOT352>());
// Object for storing the SHOT descriptors for the model.
pcl::PointCloud<pcl::SHOT352>::Ptr modelDescriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read the already computed descriptors from disk.
if (pcl::io::loadPCDFile<pcl::SHOT352>(argv[1], *sceneDescriptors) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::SHOT352>(argv[2], *modelDescriptors) != 0)
{
return -1;
}

// A kd-tree object that uses the FLANN library for fast search of nearest neighbors.
pcl::KdTreeFLANN<pcl::SHOT352> matching;
matching.setInputCloud(modelDescriptors);
// A Correspondence object stores the indices of the query and the match,
// and the distance/weight.
pcl::CorrespondencesPtr correspondences(new pcl::Correspondences());

// Check every descriptor computed for the scene.
for (size_t i = 0; i < sceneDescriptors->size(); ++i)
{
std::vector<int> neighbors(1);
std::vector<float> squaredDistances(1);
// Ignore NaNs.
if (pcl_isfinite(sceneDescriptors->at(i).descriptor[0]))
{
// Find the nearest neighbor (in descriptor space)...
int neighborCount = matching.nearestKSearch(sceneDescriptors->at(i), 1, neighbors, squaredDistances);
// ...and add a new correspondence if the distance is less than a threshold
// (SHOT distances are between 0 and 1, other descriptors use different metrics).
if (neighborCount == 1 && squaredDistances[0] < 0.25f)
{
pcl::Correspondence correspondence(neighbors[0], static_cast<int>(i), squaredDistances[0]);
correspondences->push_back(correspondence);
}
}
}
std::cout << "Found " << correspondences->size() << " correspondences." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_correspondence.html pcl::Correspondence]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_matching.tar.gz Download]
</div>

==Correspondence grouping==

Right now, all we have is a list of correspondences between keypoints in the scene and keypoints from some object(s) in the database. This does not necessarily mean that a given object is present in the scene. Consider this example: a box with two keypoints located on opposite corners. The distance between the points is known. The matching stage has found two keypoints in the scene with very similar descriptors, but as it turns out, the distance between them is way different. Unless we are taking non-rigid transformations into account (such as scaling), then we have to discard the idea that those points belong to the object.

This check can be implemented with an additional step called ''correspondence grouping''. Like the name states, it groups correspondences that are geometrically consistent (for the given object model) in clusters, and discards the ones that do not. Rotations and translations are permitted, but anything other than that will not meet the criteria. Also, as a minimum of 3 correspondences are required for retrieving the 6 DoF pose, groups with less than that can be ignored.

PCL offers a couple of classes to perform correspondence grouping.

===Geometric consistency===

This is the simplest method. It just iterates over all feature correspondences not yet grouped, and adds them to the current subset if they are geometrically consistent. The set resolution can be tuned to make the algorithm more or less forgiving when enforcing the consistency. For every subset (which should correspond with an instance of the model), this PCL class also computes the transformation (rotation and translation), making the next step ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_(pipeline)#Pose_estimation|pose estimation]]) unnecessary.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/correspondence.h>
#include <pcl/recognition/cg/geometric_consistency.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the keypoints of the scene and the model.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the unclustered and clustered correspondences.
pcl::CorrespondencesPtr correspondences(new pcl::Correspondences());
std::vector<pcl::Correspondences> clusteredCorrespondences;
// Object for storing the transformations (rotation plus translation).
std::vector<Eigen::Matrix4f, Eigen::aligned_allocator<Eigen::Matrix4f> > transformations;

// Read the keypoints from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sceneKeypoints) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *modelKeypoints) != 0)
{
return -1;
}

// Note: here you would compute the correspondences.
// It has been omitted here for simplicity.

// Object for correspondence grouping.
pcl::GeometricConsistencyGrouping<pcl::PointXYZ, pcl::PointXYZ> grouping;
grouping.setSceneCloud(sceneKeypoints);
grouping.setInputCloud(modelKeypoints);
grouping.setModelSceneCorrespondences(correspondences);
// Minimum cluster size. Default is 3 (as at least 3 correspondences
// are needed to compute the 6 DoF pose).
grouping.setGCThreshold(3);
// Resolution of the consensus set used to cluster correspondences together,
// in metric units. Default is 1.0.
grouping.setGCSize(0.01);

grouping.recognize(transformations, clusteredCorrespondences);

std::cout << "Model instances found: " << transformations.size() << std::endl << std::endl;
for (size_t i = 0; i < transformations.size(); i++)
{
std::cout << "Instance " << (i + 1) << ":" << std::endl;
std::cout << "\tHas " << clusteredCorrespondences[i].size() << " correspondences." << std::endl << std::endl;

Eigen::Matrix3f rotation = transformations[i].block<3, 3>(0, 0);
Eigen::Vector3f translation = transformations[i].block<3, 1>(0, 3);
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(0, 0), rotation(0, 1), rotation(0, 2));
printf("\t\tR = | %6.3f %6.3f %6.3f | \n", rotation(1, 0), rotation(1, 1), rotation(1, 2));
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(2, 0), rotation(2, 1), rotation(2, 2));
std::cout << std::endl;
printf("\t\tt = < %0.3f, %0.3f, %0.3f >\n", translation(0), translation(1), translation(2));
}
}</syntaxhighlight>

If you do not need to compute the transformations (or prefer to do it later in its own step), you can use the <span style="color:#FF1493">"cluster()"</span> function instead of the <span style="color:#FF1493">"recognize()"</span> one, which only takes one parameter (the object for the clustered correspondences).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (model), Points (scene), Correspondences, Minimum cluster size, Consensus set resolution
* '''Output''': Clustered correspondences, [Transformation]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_geometric_consistency_grouping.html pcl::GeometricConsistencyGrouping]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_GC_grouping.tar.gz Download]
</div>

===Hough voting===

This method is a bit more complex. It uses a technique known as [https://en.wikipedia.org/wiki/Hough_transform Hough transform], which was originally devised to perform line detection on 2D images, though it was later expanded to work with arbitrary shapes or higher dimensions. The method works with a voting procedure, with votes being cast on candidates that are found to be better suitable. If enough votes are accumulated for a given position in the space, then the shape is considered "found" there and its parameters are retrieved.

The voting procedure is carried out in a parameter space, which would have 6 dimensions in case a full pose needed to be estimated (rotation plus translation). Because that would be computationally expensive, the method implemented in PCL uses only a 3D Hough space, and employs local Reference Frames (RFs) to account for the remaining 3 DoF. Its computation can then be divided in two steps.

[[Image:Hough_voting_pipeline.png|thumb|center|600px|Use of the Hough voting scheme (red) in a 3D object recognition pipeline (image from [http://vision.deis.unibo.it/fede/papers/psivt10.pdf original paper]).]]

====Computing Reference Frames====

For every pair of correspondences, a local Reference Frame must be computed to retrieve the pose later. PCL offers a class that implements the BOrder Aware Repeatable Directions (BOARD) algorithm.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/board.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneCloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelCloud(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr sceneNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::Normal>::Ptr modelNormals(new pcl::PointCloud<pcl::Normal>);
// Objects for storing the keypoints.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the Reference Frames.
pcl::PointCloud<pcl::ReferenceFrame>::Ptr sceneRF(new pcl::PointCloud<pcl::ReferenceFrame>);
pcl::PointCloud<pcl::ReferenceFrame>::Ptr modelRF(new pcl::PointCloud<pcl::ReferenceFrame>);

// Read the scene and model clouds from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sceneCloud) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *modelCloud) != 0)
{
return -1;
}

// Note: here you would estimate the normals for the whole cloud, and choose
// the keypoints. It has been omitted here for simplicity.

// BOARD RF estimation object.
pcl::BOARDLocalReferenceFrameEstimation<pcl::PointXYZ, pcl::Normal, pcl::ReferenceFrame> rf;
// Search radius (maximum distance of the points used to estimate the X and Y axes
// of the BOARD Reference Frame for a given point).
rf.setRadiusSearch(0.02f);
// Check if support is complete, or has missing regions because it is too close to mesh borders.
rf.setFindHoles(true);

rf.setInputCloud(sceneKeypoints);
rf.setInputNormals(sceneNormals);
rf.setSearchSurface(sceneCloud);
rf.compute(*sceneRF);

rf.setInputCloud(modelKeypoints);
rf.setInputNormals(modelNormals);
rf.setSearchSurface(modelCloud);
rf.compute(*modelRF);
}</syntaxhighlight>

Be sure to check the API reference because the BOARD estimation object has many more parameters to tune.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search radius, [Tangent radius], [Margin array size], [Margin threshold], [Find holes], [Hole size threshold], [Steepness threshold]
* '''Output''': Reference Frames
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''Publication''':
** [http://vision.deis.unibo.it/LRF/LRF_repeatability_ICCV2011.pdf On the repeatability of the local reference frame for partial shape matching] (Alioscia Petrelli and Luigi Di Stefano, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_b_o_a_r_d_local_reference_frame_estimation.html pcl::BOARDLocalReferenceFrameEstimation]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_Hough_RFs.tar.gz Download]
</div>

====Clustering====

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/recognition/cg/hough_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the keypoints of the scene and the model.
pcl::PointCloud<pcl::PointXYZ>::Ptr sceneKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr modelKeypoints(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the Reference Frames.
pcl::PointCloud<pcl::ReferenceFrame>::Ptr sceneRF(new pcl::PointCloud<pcl::ReferenceFrame>);
pcl::PointCloud<pcl::ReferenceFrame>::Ptr modelRF(new pcl::PointCloud<pcl::ReferenceFrame>);
// Objects for storing the unclustered and clustered correspondences.
pcl::CorrespondencesPtr correspondences(new pcl::Correspondences());
std::vector<pcl::Correspondences> clusteredCorrespondences;
// Object for storing the transformations (rotation plus translation).
std::vector<Eigen::Matrix4f, Eigen::aligned_allocator<Eigen::Matrix4f> > transformations;

// Read the keypoints from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sceneKeypoints) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *modelKeypoints) != 0)
{
return -1;
}

// Note: here you would compute the correspondences and the Reference Frames.
// It has been omitted here for simplicity.

// Object for correspondence grouping.
pcl::Hough3DGrouping<pcl::PointXYZ, pcl::PointXYZ, pcl::ReferenceFrame, pcl::ReferenceFrame> grouping;
grouping.setInputCloud(modelKeypoints);
grouping.setInputRf(modelRF);
grouping.setSceneCloud(sceneKeypoints);
grouping.setSceneRf(sceneRF);
grouping.setModelSceneCorrespondences(correspondences);
// Minimum cluster size. Default is 3 (as at least 3 correspondences
// are needed to compute the 6 DoF pose).
grouping.setHoughThreshold(3);
// Size of each bin in the Hough space.
grouping.setHoughBinSize(3);
// If true, the vote casting procedure will interpolate the score
// between neighboring bins in the Hough space.
grouping.setUseInterpolation(true);
// If true, the vote casting procedure will use the correspondence's
// weighted distance to compute the Hough voting score.
grouping.setUseDistanceWeight(false);

grouping.recognize(transformations, clusteredCorrespondences);

std::cout << "Model instances found: " << transformations.size() << std::endl << std::endl;
for (size_t i = 0; i < transformations.size(); i++)
{
std::cout << "Instance " << (i + 1) << ":" << std::endl;
std::cout << "\tHas " << clusteredCorrespondences[i].size() << " correspondences." << std::endl << std::endl;

Eigen::Matrix3f rotation = transformations[i].block<3, 3>(0, 0);
Eigen::Vector3f translation = transformations[i].block<3, 1>(0, 3);
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(0, 0), rotation(0, 1), rotation(0, 2));
printf("\t\tR = | %6.3f %6.3f %6.3f | \n", rotation(1, 0), rotation(1, 1), rotation(1, 2));
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(2, 0), rotation(2, 1), rotation(2, 2));
std::cout << std::endl;
printf("\t\tt = < %0.3f, %0.3f, %0.3f >\n", translation(0), translation(1), translation(2));
}
}</syntaxhighlight>

If you do not give the Reference Frames to the Hough grouping object, it will calculate them itself, but then you need to specify additional parameters. See the API for more details. Also, you can <span style="color:#FF1493">"cluster()"</span> instead of <span style="color:#FF1493">"recognize()"</span> if you want to skip the pose estimation now.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (model), Points (scene), Correspondences, Reference Frames (model), Reference Frames (scene), Hough bin size, Hough threshold, [Use distance weight], [Use interpolation]
* '''Output''': Clustered correspondences, [Transformation]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/correspondence_grouping.php 3D Object Recognition based on Correspondence Grouping]
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/psivt10.pdf Object recognition in 3D scenes with occlusions and clutter by Hough voting] (Federico Tombari and Luigi Di Stefano, 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_hough3_d_grouping.html pcl::Hough3DGrouping]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_Hough_grouping.tar.gz Download]
</div>

==Pose estimation==

The two correspondence grouping classes included in PCL already compute the full pose, but you can do it manually if you want, or if you want to refine the results of the previous step (removing correspondences that are not compatible with the same 6 DoF pose). You can use a method like RANSAC (RANdom SAmple Consensus), to get the rotation and translation of the rigid transformation that best fits the correspondences of a model instance. In a previous tutorial we mentioned that there was a technique called [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration|feature-based registration]] which was an alternative to ICP, and this is it.

Of course, PCL has some classes for this. The one we are going to see adds a prerejection step to the RANSAC loop, in order to discard hypotheses that are probably wrong. To do this, the algorithm forms polygons with the points of the input sets, and then checks pose-invariant geometrical constraints. To be precise, it makes sure that the polygon edges are of similar length. You can let it run for a specified number of iterations (as, like you already know, RANSAC never actually converges), or set a threshold.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/shot.h>
#include <pcl/registration/sample_consensus_prerejective.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the scene and the model.
pcl::PointCloud<pcl::PointXYZ>::Ptr scene(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr model(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr alignedModel(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the SHOT descriptors for the scene and model.
pcl::PointCloud<pcl::SHOT352>::Ptr sceneDescriptors(new pcl::PointCloud<pcl::SHOT352>());
pcl::PointCloud<pcl::SHOT352>::Ptr modelDescriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read the clouds from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *scene) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *model) != 0)
{
return -1;
}

// Note: here you would compute or load the descriptors for both
// the scene and the model. It has been omitted here for simplicity.

// Object for pose estimation.
pcl::SampleConsensusPrerejective<pcl::PointXYZ, pcl::PointXYZ, pcl::SHOT352> pose;
pose.setInputSource(model);
pose.setInputTarget(scene);
pose.setSourceFeatures(modelDescriptors);
pose.setTargetFeatures(sceneDescriptors);
// Instead of matching a descriptor with its nearest neighbor, choose randomly between
// the N closest ones, making it more robust to outliers, but increasing time.
pose.setCorrespondenceRandomness(2);
// Set the fraction (0-1) of inlier points required for accepting a transformation.
// At least this number of points will need to be aligned to accept a pose.
pose.setInlierFraction(0.25f);
// Set the number of samples to use during each iteration (minimum for 6 DoF is 3).
pose.setNumberOfSamples(3);
// Set the similarity threshold (0-1) between edge lengths of the polygons. The
// closer to 1, the more strict the rejector will be, probably discarding acceptable poses.
pose.setSimilarityThreshold(0.6f);
// Set the maximum distance threshold between two correspondent points in source and target.
// If the distance is larger, the points will be ignored in the alignment process.
pose.setMaxCorrespondenceDistance(0.01f);

pose.align(*alignedModel);

if (pose.hasConverged())
{
Eigen::Matrix4f transformation = pose.getFinalTransformation();
Eigen::Matrix3f rotation = transformation.block<3, 3>(0, 0);
Eigen::Vector3f translation = transformation.block<3, 1>(0, 3);

std::cout << "Transformation matrix:" << std::endl << std::endl;
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(0, 0), rotation(0, 1), rotation(0, 2));
printf("\t\tR = | %6.3f %6.3f %6.3f | \n", rotation(1, 0), rotation(1, 1), rotation(1, 2));
printf("\t\t | %6.3f %6.3f %6.3f | \n", rotation(2, 0), rotation(2, 1), rotation(2, 2));
std::cout << std::endl;
printf("\t\tt = < %0.3f, %0.3f, %0.3f >\n", translation(0), translation(1), translation(2));
}
else std::cout << "Did not converge." << std::endl;
}</syntaxhighlight>

For an alternative, check the [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_registration.html pcl::SampleConsensusModelRegistration] or the [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_initial_alignment.html pcl::SampleConsensusInitialAlignment] classes.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (model), Points (scene), Descriptors (model), Descriptors (scene), [Correspondence randomness], [Inlier fraction], [Number of samples], [Similarity threshold]
* '''Output''': Aligned model, [Transformation]
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/alignment_prerejective.php Robust pose estimation of rigid objects]
** [http://pointclouds.org/documentation/tutorials/template_alignment.php Aligning object templates to a point cloud]
* '''Publication''':
** [http://personal.lut.fi/users/joni.kamarainen/downloads/publications/icra2013.pdf Pose Estimation using Local Structure-Specific Shape and Appearance Context] (Anders Glent Buch et al., 2013)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_prerejective.html pcl::SampleConsensusPrerejective]
** [http://docs.pointclouds.org/trunk/classpcl_1_1registration_1_1_correspondence_rejector_poly.html pcl::registration::CorrespondenceRejectorPoly]
* [http://robotica.unileon.es/~victorm/PCL_local_pipeline_pose.tar.gz Download]
</div>

=Global pipeline=

Because of the way global descriptors work, the global pipeline for object recognition differs in some steps.

==Segmentation==

Unlike local descriptors, global ones understand the notion of ''object''. They are not computed for single points, but for whole clusters. Because of this, the first step is to perform [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]] on the cloud in order to retrieve all objects. You can use any method you like from the ones available in PCL, as long as it works for you. Also, performing some kind of preprocessing is a good idea, like removing all clusters that are smaller or bigger than certain thresholds, which should be set according to the objects in the database. Yet another possibility is to perform [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Plane_model|plane segmentation]] to find any table(s) in the scene, and consider only clusters sitting on it (if we can assume there is a table, of course).

PCL provides the [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_polygonal_prism_data.html pcl::ExtractPolygonalPrismData] class for the latter task. By giving a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Convex_hull|convex hull]] computed from the plane coefficients, it will extrude it a certain height to create a prism, and then give back all points that lie inside. This is the code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/surface/convex_hull.h>
#include <pcl/segmentation/extract_polygonal_prism_data.h>
#include <pcl/visualization/cloud_viewer.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr plane(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr convexHull(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr objects(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Get the plane model, if present.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setOptimizeCoefficients(true);
pcl::PointIndices::Ptr planeIndices(new pcl::PointIndices);
segmentation.segment(*planeIndices, *coefficients);

if (planeIndices->indices.size() == 0)
std::cout << "Could not find a plane in the scene." << std::endl;
else
{
// Copy the points of the plane to a new cloud.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloud);
extract.setIndices(planeIndices);
extract.filter(*plane);

// Retrieve the convex hull.
pcl::ConvexHull<pcl::PointXYZ> hull;
hull.setInputCloud(plane);
// Make sure that the resulting hull is bidimensional.
hull.setDimension(2);
hull.reconstruct(*convexHull);

// Redundant check.
if (hull.getDimension() == 2)
{
// Prism object.
pcl::ExtractPolygonalPrismData<pcl::PointXYZ> prism;
prism.setInputCloud(cloud);
prism.setInputPlanarHull(convexHull);
// First parameter: minimum Z value. Set to 0, segments objects lying on the plane (can be negative).
// Second parameter: maximum Z value, set to 10cm. Tune it according to the height of the objects you expect.
prism.setHeightLimits(0.0f, 0.1f);
pcl::PointIndices::Ptr objectIndices(new pcl::PointIndices);

prism.segment(*objectIndices);

// Get and show all points retrieved by the hull.
extract.setIndices(objectIndices);
extract.filter(*objects);
pcl::visualization::CloudViewer viewerObjects("Objects on table");
viewerObjects.showCloud(objects);
while (!viewerObjects.wasStopped())
{
// Do nothing but wait.
}
}
else std::cout << "The chosen hull is not planar." << std::endl;
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:Prism_scene_before.png | Original scene, with a mug on a table.
File:Prism_scene_after.png | Result after using the polygonal prism class.
</gallery></center>

If you get some residual points of the table in the output, try increasing the minimum Z value a bit.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Convex hull, Height limits, [Dimensionality]
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_polygonal_prism_data.html pcl::ExtractPolygonalPrismData]
* [http://robotica.unileon.es/~victorm/PCL_polygonal_prism.tar.gz Download]
</div>

==Computing descriptors==

For every cluster that has survived the previous step, a global descriptor must be computed. Most only output one histogram (except for CVFH and derivatives), so the database will be smaller than with local descriptors. Check the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Table|list]] and choose the one that fits you best.

===CRH===

In a previous article about global descriptors we talked about how they were invariant to the camera roll angle. When [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#CVFH|CVFH]] was presented, the computation of a Camera Roll Histogram (CRH) was proposed to overcome this. The CRH should be computed and stored next to the global descriptor (VFH, CVFH...) for its use in a later phase, the pose estimation.

The CRH is computed as follows: for every point, its normal is projected onto a plane orthogonal to the vector given by the camera center and the centroid of the cluster (VFH) or region (CVFH). Then, the angle of the projected normal to the up vector of the camera is computed and added to the histogram, which has 90 bins (for a resolution of 4º). The projected normals are weighted by their magnitudes to reduce the effect of input noise. Check the [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 paper] for details.

You can compute it using PCL, of course:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/common/centroid.h>
#include <pcl/features/crh.h>

// A handy typedef.
typedef pcl::Histogram<90> CRH90;

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CRH.
pcl::PointCloud<CRH90>::Ptr histogram(new pcl::PointCloud<CRH90>);

// Note: this cloud file should contain a snapshot of the object. Remember
// that you need to compute a CRH for every VFH or CVFH descriptor that you
// are going to have (that is, one for every snapshot).

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CRH estimation object.
pcl::CRHEstimation<pcl::PointXYZ, pcl::Normal, CRH90> crh;
crh.setInputCloud(object);
crh.setInputNormals(normals);
Eigen::Vector4f centroid;
pcl::compute3DCentroid(*object, centroid);
crh.setCentroid(centroid);

// Compute the CRH.
crh.compute(*histogram);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Centroid
* '''Output''': Camera roll histogram
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_estimation.html pcl::CRHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CRH.tar.gz Download]
</div>

==Matching==

Like with the local pipeline, once all the descriptors have been computed we have to perform a search for their nearest neighbors in the database. The [https://en.wikipedia.org/wiki/Taxicab_geometry Manhattan or L1 distance] is recommended over the [https://en.wikipedia.org/wiki/Euclidean_distance Euclidean or L2] one because it is more robust to occlusions.

Usually, more than one neighbor is retrieved, which means that the found object's pose is somewhere between the two. The next steps can help improving the accuraccy of the estimation.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_recognition.php Cluster Recognition and 6DOF Pose Estimation using VFH descriptors]
</div>

==Pose estimation==

The object's position can be determined by computing and aligning the [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Computing_the_centroid|centroids]] of the clusters (the one in the scene, and the one in the snapshot). For the rotation, we must use the viewpoint information encoded in the descriptor (you can check the ground truth pose information that was saved with the matched snapshot). This still leaves the roll angle unresolved, as global descriptors are invariant to it. Now is when the Camera Roll Histogram ([[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#CRH|CRH]]) that we should have stored next to the descriptor comes in handy. The angle can be obtained by aligning the current CRH with the stored one, completing the 6 DoF pose estimation.

===CRH===

This is the code for retrieving the roll angle using CRH:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>
#include <pcl/features/crh.h>
#include <pcl/recognition/crh_alignment.h>

#include <iostream>
#include <vector>

// A handy typedef.
typedef pcl::Histogram<90> CRH90;

int
main(int argc, char** argv)
{
// Clouds for storing the object's cluster and view.
pcl::PointCloud<pcl::PointXYZ>::Ptr viewCloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr clusterCloud(new pcl::PointCloud<pcl::PointXYZ>);
// Objects for storing the CRHs of both.
pcl::PointCloud<CRH90>::Ptr viewCRH(new pcl::PointCloud<CRH90>);
pcl::PointCloud<CRH90>::Ptr clusterCRH(new pcl::PointCloud<CRH90>);
// Objects for storing the centroids.
Eigen::Vector4f viewCentroid;
Eigen::Vector4f clusterCentroid;

// Read the trained view and the scene cluster from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *viewCloud) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *clusterCloud) != 0)
{
return -1;
}

// Note: here you would compute the CRHs and centroids of both clusters.
// It has been omitted here for simplicity.

// CRH alignment object.
pcl::CRHAlignment<pcl::PointXYZ, 90> alignment;
alignment.setInputAndTargetView(clusterCloud, viewCloud);
// CRHAlignment works with Vector3f, not Vector4f.
Eigen::Vector3f viewCentroid3f(viewCentroid[0], viewCentroid[1], viewCentroid[2]);
Eigen::Vector3f clusterCentroid3f(clusterCentroid[0], clusterCentroid[1], clusterCentroid[2]);
alignment.setInputAndTargetCentroids(clusterCentroid3f, viewCentroid3f);

// Compute the roll angle(s).
std::vector<float> angles;
alignment.computeRollAngle(*clusterCRH, *viewCRH, angles);

if (angles.size() > 0)
{
std::cout << "List of angles where the histograms correlate:" << std::endl;

for (int i = 0; i < angles.size(); i++)
{
std::cout << "\t" << angles.at(i) << " degrees." << std::endl;
}
}
}</syntaxhighlight>

This will return a list of most probable roll angles. If you want, you can instead use the [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_alignment.html#a32c40356d56e22d48eccce5e27e436f2 align()] function, which calls <span style="color:#FF1493">"computeRollAngle()"</span> internally. It will return a list of transformations that include the translation (computed from the difference between the centroid's coordinates) and the roll angles, but not the yaw or pitch (you have to get those with pose estimation).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (input and target), Centroids (input and target), CRHs (input and target)
* '''Output''': Roll angles, [Transformations]
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_r_h_alignment.html pcl::CRHAlignment]
* [http://robotica.unileon.es/~victorm/PCL_roll_estimation.tar.gz Download]
</div>

=Postprocessing=

The pipelines, as they are, should already return a decent result, but we can implement optional steps to improve the accuraccy of the pose estimation, and decrease the probability of a false positive.

==Pose refinement==

The 6 DoF pose estimation can be refined with the use of the [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Iterative_Closest_Point_.28ICP.29|Iterative Closest Point]] (ICP) algorithm. It will try to continuously realign the clouds to improve the transformation, until the termination condition is met (maximum iterations, and distance or error threshold). This is identical to performing [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Registration|registration]], only between input and model.

==Hypothesis verification==

''(Work in progress)''

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 4: 3D object recognition (descriptors)

2015-11-01T23:12:44Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

It is time to learn the basics of one of the most interesting applications of point cloud processing: 3D object recognition. Akin to 2D recognition, this technique relies on finding good keypoints (characteristic points) in the cloud, and matching them to a set of previously saved ones. But 3D has several advantages over 2D: namely, we will be able to estimate with decent accuracy the exact position and orientation of the object, relative to the sensor. Also, 3D object recognition tends to be more robust to clutter (crowded scenes where objects in the front occluding objects in the background). And finally, having information about the object's shape will help with collision avoidance or grasping operations.

In this first tutorial we will see what descriptors are, how many types are there available in PCL, and how to compute them.

=Overview=

The basis of 3D object recognition is to find a set of correspondences between two different clouds, one of them containing the object we are looking for. In order to do this, we need a way to compare points in an unambiguous manner. Until now, we have worked with points that store the XYZ coordinates, the RGB color... but none of those properties are unique enough. In two sequential scans, two points could share the same coordinates despite belonging to different surfaces, and using the color information takes us back to 2D recognition, will all the lightning related problems.

In a previous tutorial, we talked about [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Feature_estimation | features]], before introducing the normals. Normals are an example of feature, because they encode information about the vicinity of the point. That is, the neighboring points are taken into account when computing it, giving us an idea of how the surrounding surface is. But this is not enough. For a feature to be optimal, it must meet the following criteria:

# '''It must be robust to transformations''': rigid transformations (the ones that do not change the distance between points) like translations and rotations must not affect the feature. Even if we play with the cloud a bit beforehand, there should be no difference.
# '''It must be robust to noise''': measurement errors that cause noise should not change the feature estimation much.
# '''It must be resolution invariant''': if sampled with different density (like after performing downsampling), the result must be identical or similar.

This is where descriptors come in. They are more complex (and precise) signatures of a point, that encode a lot of information about the surrounding geometry. The purpose is to unequivocally identify a point across multiple point clouds, no matter the noise, resolution or transformations. Also, some of them capture additional data about the object they belong to, like the viewpoint (that lets us retrieve the pose).

[[Image:Correspondence.jpg|thumb|center|600px|Finding correspondences between point features of two clouds (image from http://pointclouds.org/).]]

There are many 3D descriptors implemented into PCL. Each one has its own method for computing unique values for a point. Some use the difference between the angles of the normals of the point and its neighbors, for example. Others use the distances between the points. Because of this, some are inherently better or worse for certain purposes. A given descriptor may be scale invariant, and another one may be better with occlusions and partial views of objects. Which one you choose depends on what you want to do.

After calculating the necessary values, an additional step is performed to reduce the descriptor size: the result is binned into an [https://en.wikipedia.org/wiki/Histogram histogram]. To do this, the value range of each variable that makes up the descriptor is divided into ''n'' subdivisions, and the number of occurrences in each one is counted. Try to imagine a descriptor that computes a single variable, that ranges from 1 to 100. We choose to create 10 bins for it, so the first bin would gather all occurrences between 1 and 10, the second from 11 to 20, and so on. We look at the value of the variable for the first point-neighbor pair, and it is 27, so we increment the value of the third bin by 1. We keep doing this until we get a final histogram for that keypoint. The bin size must be carefully chosen depending on how descriptive that variable is (the variables do not have to share the same number of bins, and also the bins do not have to be of the same size; if for example most values from the previous example fell in the 50-100 range then it would be sensible to have more bins of smaller size in that range).

Descriptors can be classified in two main categories: global and local. The process for computing and using each one (recognition pipeline) is different, so each will be explained in its own section in this article.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/how_features_work.php How 3D Features work in PCL]
** [https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features Overview and Comparison of Features]
** [http://www.pointclouds.org/assets/icra2013/pcl_features_icra13.pdf How does a good feature look like?]
* '''Publication''':
** [https://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf Point Cloud Library: Three-Dimensional Object Recognition and 6 DoF Pose Estimation] (Aitor Aldoma et al., 2012)
</div>

==Table==

The following table will give you a hint of how many descriptors are there in PCL, and some of their features:

{| class="wikitable sortable" style="margin: auto; text-align:center" cellpadding="15"
|+ 3D descriptors
|- style="vertical-align:middle;"
! scope="col"| Name
! scope="col"| Type
! scope="col"| Size<sup>†</sup>
! scope="col"| Custom PointType<sup>††</sup>
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#PFH|PFH]] (Point Feature Histogram)
| style="color: sienna;" | Local
| 125
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#FPFH|FPFH]] (Fast Point Feature Histogram)
| style="color: sienna;" | Local
| 33
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RSD|RSD]] (Radius-Based Surface Descriptor)
| style="color: sienna;" | Local
| 289
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#3DSC|3DSC]] (3D Shape Context)
| style="color: sienna;" | Local
| 1980
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#USC|USC]] (Unique Shape Context)
| style="color: sienna;" | Local
| 1960
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#SHOT|SHOT]] (Signatures of Histograms of Orientations)
| style="color: sienna;" | Local
| 352
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#Spin_image|Spin image]]
| style="color: sienna;" | Local
| 153*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RIFT|RIFT]] (Rotation-Invariant Feature Transform)
| style="color: sienna;" | Local
| 32*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#NARF|NARF]] (Normal Aligned Radial Feature)
| style="color: sienna;" | Local
| 36
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#RoPS|RoPS]] (Rotational Projection Statistics)
| style="color: sienna;" | Local
| 135*
| style="color: red;" | No
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#VFH|VFH]] (Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#CVFH|CVFH]] (Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#OUR-CVFH|OUR-CVFH]] (Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram)
| style="color: darkcyan;" | Global
| 308
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#ESF|ESF]] (Ensemble of Shape Functions)
| style="color: darkcyan;" | Global
| 640
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GFPFH|GFPFH]] (Global Fast Point Feature Histogram)
| style="color: darkcyan;" | Global
| 16
| style="color: green;" | Yes
|- style="vertical-align:middle;"
| [[PCL/OpenNI_tutorial_4:_3D_object_recognition_(descriptors)#GRSD|GRSD]] (Global Radius-Based Surface Descriptor)
| style="color: darkcyan;" | Global
| 21
| style="color: green;" | Yes
|}

† Values marked with an asterisk (*) indicate that the descriptor's size depends on some parameter(s), and the one given is for the default values.

†† Descriptors without their own custom PointType use the generic <span style="color:#FF1493">"pcl::Histogram<>"</span> type. See [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Saving_and_loading|Saving and loading]].

Optionally, you can download a document with a less simple version of the table. Page format is A4, landscape:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Downloads''':
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.odt Table as original ODT] (uses font embedding, requires LibreOffice 4)
** [http://robotica.unileon.es/~victorm/3D_descriptors_comparison.pdf Table as PDF]
</div>

=Local descriptors=

Local descriptors are computed for individual points that we give as input. They have no notion of what an object is, they just describe how the local geometry is around that point. Usually, it is your task to choose which points you want a descriptor to be computed for: the ''keypoints''. Most of the time, you can get away by just performing a downsampling and choosing all remaining points, but keypoint detectors are available, like the one used for [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#Finding_keypoints|NARF]], or [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#ISS|ISS]].

Local descriptors are used for object recognition and [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Registration | registration]]. Now we will see which ones are implemented into PCL.

==PFH==

PFH stands for Point Feature Histogram. It is one of the most important descriptors offered by PCL and the basis of others such as FPFH. The PFH tries to capture information of the geometry surrounding the point by analyzing the difference between the directions of the normals in the vicinity (and because of this, an imprecise normal estimation may produce low-quality descriptors).

First, the algorithm pairs all points in the vicinity (not just the chosen keypoint with its neighbors, but also the neighbors with themselves). Then, for each pair, a [https://en.wikipedia.org/wiki/Darboux_frame fixed coordinate frame] is computed from their normals. With this frame, the difference between the normals can be encoded with 3 angular variables. These variables, together with the euclidean distance between the points, are saved, and then binned to an histogram when all pairs have been computed. The final descriptor is the concatenation of the histograms of each variable (4 in total).

<center><gallery widths=300px>
File:PFH_neighbors.png | Point pairs established when computing the PFH for a point (image from http://pointclouds.org).
File:PFH_frame.png | Fixed coordinate frame and angular features computed for one of the pairs (image from http://pointclouds.org).
</gallery></center>

Computing descriptors in PCL is very easy, and the PFH is not an exception:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/pfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the PFH descriptors for each point.
pcl::PointCloud<pcl::PFHSignature125>::Ptr descriptors(new pcl::PointCloud<pcl::PFHSignature125>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// PFH estimation object.
pcl::PFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::PFHSignature125> pfh;
pfh.setInputCloud(cloud);
pfh.setInputNormals(normals);
pfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
pfh.setRadiusSearch(0.05);

pfh.compute(*descriptors);
}</syntaxhighlight>

As you can see, PCL uses the <span style="color:#FF1493">"PFHSignature125"</span> type to save the descriptor to. This means that the descriptor's size is 125 (the dimensionality of the feature vector). Dividing a feature in ''D'' dimensional space in ''B'' divisions requires a total of B<sup>D</sup> bins. The original proposal makes use of the distance between the points, but the implementation of PCL does not, as it was not considered discriminative enough (specially in 2.5D scans, where the distance between points increases the further away from the sensor). Hence, the remaining features (with one dimension each) can be divided in 5 divisions, resulting in a 125-dimensional (5<sup>3</sup>) vector.

The final object that stores the computed descriptors can be handled like a normal cloud (even saved to, or read from, disk), and it has the same number of "points" than the original one. The <span style="color:#FF1493">"PFHSignature125"</span> object at position 0, for example, stores the PFH descriptor for the <span style="color:#FF1493">"PointXYZ"</span> point at the same position in the cloud.

For additional details about the descriptor, check the original publications listed below, or PCL's tutorial.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': PFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pfh_estimation.php Point Feature Histograms (PFH) descriptors]
* '''Publications''':
** [http://ias.in.tum.de/_media/spezial/bib/rusu08ias.pdf Persistent Point Feature Histograms for 3D Point Clouds] (Radu Bogdan Rusu et al., 2008)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.4, page 50)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_p_f_h_estimation.html pcl::PFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_PFH.tar.gz Download]
</div>

===FPFH===

PFH gives accurate results, but it has a drawback: it is too computationally expensive to perform at real time. For a cloud of ''n'' keypoints with ''k'' neighbors considered, it has a [https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations complexity] of ''O(nk<sup>2</sup>)''. Because of this, a derived descriptor was created, named FPFH (Fast Point Feature Histogram).

The FPFH considers only the direct connections between the current keypoint and its neighbors, removing additional links between neighbors. This takes the complexity down to ''O(nk)''. Because of this, the resulting histogram is referred to as SPFH (Simplified Point Feature Histogram). The reference frame and the angular variables are computed as always.

[[Image:FPFH_neighbors.png|thumb|center|400px|Point pairs established when computing the FPFH for a point (image from http://pointclouds.org).]]

To account for the loss of these extra connections, an additional step takes place after all histograms have been computed: the SPFHs of a point's neighbors are "merged" with its own, weighted according to the distance. This has the effect of giving a point surface information of points as far away as 2 times the radius used. Finally, the 3 histograms (distance is not used) are concatenated to compose the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/fpfh.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the FPFH descriptors for each point.
pcl::PointCloud<pcl::FPFHSignature33>::Ptr descriptors(new pcl::PointCloud<pcl::FPFHSignature33>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// FPFH estimation object.
pcl::FPFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::FPFHSignature33> fpfh;
fpfh.setInputCloud(cloud);
fpfh.setInputNormals(normals);
fpfh.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
fpfh.setRadiusSearch(0.05);

fpfh.compute(*descriptors);
}</syntaxhighlight>

An additional implementation of the FPFH estimation that takes advantage of multithreaded optimizations (with the [https://en.wikipedia.org/wiki/OpenMP OpenMP API]) is available in the class <span style="color:#FF1493">"FPFHEstimationOMP"</span>. Its interface is identical to the standard unoptimized implementation. Using it will result in a big performance boost on multi-core systems, meaning faster computation times. Remember to include the header <span style="color:#FF1493">"fpfh_omp.h"</span> instead.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius
* '''Output''': FPFH descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/fpfh_estimation.php Fast Point Feature Histograms (FPFH) descriptors]
* '''Publications''':
** [http://files.rbrusu.com/publications/Rusu09ICRA.pdf Fast Point Feature Histograms (FPFH) for 3D Registration] (Radu Bogdan Rusu et al., 2009)
** [http://files.rbrusu.com/publications/RusuPhDThesis.pdf Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments] (Radu Bogdan Rusu, 2009, section 4.5, page 57)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation.html pcl::FPFHEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_f_p_f_h_estimation_o_m_p.html pcl::FPFHEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_FPFH.tar.gz Download]
</div>

==RSD==

The Radius-Based Surface Descriptor encodes the radial relationship of the point and its neighborhood. For every pair of the keypoint with a neighbor, the algorithm computes the distance between them, and the difference between their normals. Then, by assuming that both points lie on the surface of a sphere, said sphere is found by fitting not only the points, but also the normals (otherwise, there would be infinite possible spheres). Finally, from all the point-neighbor spheres, only the ones with the maximum and minimum radii are kept and saved to the descriptor of that point.

As you may have deduced already, when two points lie on a flat surface, the sphere radius will be infinite. If, on the other hand, they lie on the curved face of a cylinder, the radius will be more or less the same as that of the cylinder. This allows us to tell objects apart with RSD. The algorithm takes a parameter that sets the maximum radius at which the points will be considered to be part of a plane.

<center><gallery widths=300px>
File:RSD_sphere.png | Sphere that fits two points with their normals (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
File:RSD_values.png | RSD values for a cloud; points on a flat surface have maximum values (image from [http://ias.in.tum.de/_media/events/rgbd2011/marton.pdf original presentation]).
</gallery></center>

This is the code for compiling the RSD descriptor:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/rsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RSD descriptors for each point.
pcl::PointCloud<pcl::PrincipalRadiiRSD>::Ptr descriptors(new pcl::PointCloud<pcl::PrincipalRadiiRSD>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// RSD estimation object.
RSDEstimation<pcl::PointXYZ, pcl::Normal, pcl::PrincipalRadiiRSD> rsd;
rsd.setInputCloud(cloud);
rsd.setInputNormals(normals);
rsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
rsd.setRadiusSearch(0.05);
// Plane radius. Any radius larger than this is considered infinite (a plane).
rsd.setPlaneRadius(0.1);
// Do we want to save the full distance-angle histograms?
rsd.setSaveHistograms(false);

rsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

Optionally, you can use the <span style="color:#FF1493">"setSaveHistograms()"</span> function to enable the saving of the full distance-angle histograms, and then use <span style="color:#FF1493">"getHistograms()"</span> to retrieve them.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Maximum Radius, [Subdivisions], [Save full histograms]
* '''Output''': RSD descriptors
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.391.7398&rep=rep1&type=pdf General 3D Modelling of Novel Objects from a Single View] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_s_d_estimation.html pcl::RSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RSD.tar.gz Download]
</div>

==3DSC==

The 3D Shape Context is a descriptor that extends its existing 2D counterpart to the third dimension. It works by creating a support structure (a sphere, to be precise) centered at the point we are computing the descriptor for, with the given search radius. The "north pole" of that sphere (the notion of "up") is pointed to match the normal at that point. Then, the sphere is divided in 3D regions or bins. In the first 2 coordinates (azimuth and elevation) the divisions are equally spaced, but in the third (the radial dimension), divisions are logarithmically spaced so they are smaller towards the center. A minimum radius can be specified to prevent very small bins, that would be too sensitive to small changes in the surface.

[[Image:3DSC_support_structure.png|thumb|center|200px|Support structure to compute the 3DSC for a point (image from [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf original paper]).]]

For each bin, a weighted count is accumulated for every neighboring point that lies within. The weight depends on the volume of the bin and the local point density (number of points around the current neighbor). This gives the descriptor some degree of resolution invariance.

We have mentioned that the sphere is given the direction of the normal. This still leaves one degree of freedom (only two axes have been locked, the azimuth remains free). Because of this, the descriptor so far does not cope with rotation. To overcome this (so the same point in two different clouds has the same value), the support sphere is rotated around the normal ''N'' times (a number of degrees that corresponds with the divisions in the azimuth) and the process is repeated for each, giving a total of ''N'' descriptors for that point.

You can compute the 3DSC descriptor the following way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/3dsc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the 3DSC descriptors for each point.
pcl::PointCloud<pcl::ShapeContext1980>::Ptr descriptors(new pcl::PointCloud<pcl::ShapeContext1980>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// 3DSC estimation object.
pcl::ShapeContext3DEstimation<pcl::PointXYZ, pcl::Normal, pcl::ShapeContext1980> sc3d;
sc3d.setInputCloud(cloud);
sc3d.setInputNormals(normals);
sc3d.setSearchMethod(kdtree);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
sc3d.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
sc3d.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
sc3d.setPointDensityRadius(0.05 / 5.0);

sc3d.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Search method, Radius, Minimal radius, Point density radius
* '''Output''': 3DSC descriptors
* '''Publication''':
** [http://www.cs.jhu.edu/~misha/Papers/Frome04.pdf Recognizing Objects in Range Data Using Regional Point Descriptors] (Andrea Frome et al., 2004)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_shape_context3_d_estimation.html pcl::ShapeContext3DEstimation]
* [http://robotica.unileon.es/~victorm/PCL_3DSC.tar.gz Download]
</div>

===USC===

The Unique Shape Context descriptor extends the 3DSC by defining a local reference frame, in order to provide an unique orientation for each point. This not only improves the accuracy of the descriptor, it also reduces its size, as computing multiple descriptors to account for orientation is no longer necessary.

You can check the second publication listed below to learn more about how the LRF is computed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/usc.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the USC descriptors for each point.
pcl::PointCloud<pcl::UniqueShapeContext1960>::Ptr descriptors(new pcl::PointCloud<pcl::UniqueShapeContext1960>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// USC estimation object.
pcl::UniqueShapeContext<pcl::PointXYZ, pcl::UniqueShapeContext1960, pcl::ReferenceFrame> usc;
usc.setInputCloud(cloud);
// Search radius, to look for neighbors. It will also be the radius of the support sphere.
usc.setRadiusSearch(0.05);
// The minimal radius value for the search sphere, to avoid being too sensitive
// in bins close to the center of the sphere.
usc.setMinimalRadius(0.05 / 10.0);
// Radius used to compute the local point density for the neighbors
// (the density is the number of points within that radius).
usc.setPointDensityRadius(0.05 / 5.0);
// Set the radius to compute the Local Reference Frame.
usc.setLocalRadius(0.05);

usc.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk). For 1.7 and below, change UniqueShapeContext1960 to ShapeContext1980, and edit CMakeLists.txt.'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Minimal radius, Point density radius, Local radius
* '''Output''': USC descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/3dor10.pdf Unique Shape Context for 3D Data Description] (Federico Tombari et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_unique_shape_context.html pcl::UniqueShapeContext]
* [http://robotica.unileon.es/~victorm/PCL_USC.tar.gz Download]
</div>

==SHOT==

SHOT stands for Signature of Histograms of Orientations. Like 3DSC, it encodes information about the topology (surface) withing a spherical support structure. This sphere is divided in 32 bins or volumes, with 8 divisions along the azimuth, 2 along the elevation, and 2 along the radius. For every volume, a one-dimensional local histogram is computed. The variable chosen is the angle between the normal of the keypoint and the current point within that volume (to be precise, the cosine, which was found to be better suitable).

[[Image:SHOT_support_structure.png|thumb|center|200px|Support structure to compute SHOT. Only 4 azimuth divisions are shown for clarity (image from [http://vision.deis.unibo.it/fede/papers/eccv10.pdf original paper]).]]

When all local histograms have been computed, they are stitched together in a final descriptor. Like the USC descriptor, SHOT makes use of a local reference frame, making it rotation invariant. It is also robust to noise and clutter.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/shot.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the SHOT descriptors for each point.
pcl::PointCloud<pcl::SHOT352>::Ptr descriptors(new pcl::PointCloud<pcl::SHOT352>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// SHOT estimation object.
pcl::SHOTEstimation<pcl::PointXYZ, pcl::Normal, pcl::SHOT352> shot;
shot.setInputCloud(cloud);
shot.setInputNormals(normals);
// The radius that defines which of the keypoint's neighbors are described.
// If too large, there may be clutter, and if too small, not enough points may be found.
shot.setRadiusSearch(0.02);

shot.compute(*descriptors);
}</syntaxhighlight>

Like with FPFH, a multithreading-optimized variant is available with <span style="color:#FF1493">"SHOTEstimationOMP"</span>, that makes use of OpenMP. You need to include the header <span style="color:#FF1493">"shot_omp.h"</span>. Also, another variant that uses the texture for matching is available, <span style="color:#FF1493">"SHOTColorEstimation"</span>, with an optimized version too (see the second publication for more details). It outputs a <span style="color:#FF1493">"SHOT1344"</span> descriptor.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius
* '''Output''': SHOT descriptors
* '''Publications''':
** [http://vision.deis.unibo.it/fede/papers/eccv10.pdf Unique Signatures of Histograms for Local Surface Description] (Federico Tombari et al., 2010)
** [http://www.vision.deis.unibo.it/fede/papers/icip11.pdf A Combined Texture-Shaped Descriptor for Enhanced 3D Feature Matching] (Federico Tombari et al., 2011)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation.html pcl::SHOTEstimation],
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation.html pcl::SHOTColorEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_estimation_o_m_p.html pcl::SHOTEstimationOMP]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_s_h_o_t_color_estimation_o_m_p.html pcl::SHOTColorEstimationOMP]
* [http://robotica.unileon.es/~victorm/PCL_SHOT.tar.gz Download]
</div>

==Spin image==

The Spin Image (SI) is the oldest descriptor we are going to see here. It has been around since 1997, but it still sees some use for certain applications. It was originally designed to describe surfaces made by vertices, edges and polygons, but it has been since adapted for point clouds. The descriptor is unlike all others in that the output resembles an image that can be compared with another with the usual means.

The support structure used is a cylinder, centered at the point, with a given radius and height, and aligned with the normal. This cylinder is divided radially and vertically into volumes. For each one, the number of neighbors lying inside is added up, eventually producing a descriptor. Weighting and interpolation are used to improve the result. The final descriptor can be seen as a grayscale image where dark areas correspond to volumes with higher point density.

[[Image:Spin images.png|thumb|center|400px|Spin images computed for 3 points of a model (image from [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf original thesis]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/spin_image.h>

// A handy typedef.
typedef pcl::Histogram<153> SpinImage;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the spin image for each point.
pcl::PointCloud<SpinImage>::Ptr descriptors(new pcl::PointCloud<SpinImage>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Spin image estimation object.
pcl::SpinImageEstimation<pcl::PointXYZ, pcl::Normal, SpinImage> si;
si.setInputCloud(cloud);
si.setInputNormals(normals);
// Radius of the support cylinder.
si.setRadiusSearch(0.02);
// Set the resolution of the spin image (the number of bins along one dimension).
// Note: you must change the output histogram size to reflect this.
si.setImageWidth(8);

si.compute(*descriptors);
}</syntaxhighlight>

The Spin Image estimation object provides more methods for tuning the estimation, so checking the API is recommended.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Normals, Radius, Image resolution
* '''Output''': Spin images
* '''Publications''':
** [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4190&rep=rep1&type=pdf Spin-Images: A Representation for 3-D Surface Matching] (Andrew Edie Johnson, 1997)
** [http://www.cs.jhu.edu/~misha/Papers/Johnson99.pdf Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes] (Andrew Edie Johnson and Martial Hebert, 1999)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_spin_image_estimation.html pcl::SpinImageEstimation]
* [http://robotica.unileon.es/~victorm/PCL_spin_image.tar.gz Download]
</div>

==RIFT==

The Rotation-Invariant Feature Transform, like the spin image, takes some concepts from 2D features, in this case from the Scale-Invariant Feature Transform ([https://en.wikipedia.org/wiki/Scale-invariant_feature_transform SIFT]). It is the only descriptor seen here that requires intensity information in order to compute it (it can be obtained from the RGB color values). This means, of course, that you will not be able to use RIFT with standard XYZ clouds, you also need the texture.

In the first step, a circular patch (with the given radius) is fitted on the surface the point lies on. This patch is divided into concentric rings, according to the chosen distance bin size. Then, an histogram is populated with all the point's neighbors lying inside a sphere centered at that point and with the mentioned radius. The distance and the orientation of the intensity gradient at each point are considered. To make it rotation invariant, the angle between the gradient orientation and the vector pointing outward from the center of the patch is measured.

[[Image:RIFT.png|thumb|center|600px|RIFT feature values at 3 different locations in the descriptor (image from [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf original paper]).]]

The authors' original implementation uses 4 rings and 8 histogram orientations, which produce a descriptor of size 32. RIFT is not robust to texture flipping, though this was never considered a big issue.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/intensity_gradient.h>
#include <pcl/features/rift.h>

// A handy typedef.
typedef pcl::Histogram<32> RIFT32;

int
main(int argc, char** argv)
{
// Object for storing the point cloud with color information.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloudColor(new pcl::PointCloud<pcl::PointXYZRGB>);
// Object for storing the point cloud with intensity value.
pcl::PointCloud<pcl::PointXYZI>::Ptr cloudIntensity(new pcl::PointCloud<pcl::PointXYZI>);
// Object for storing the intensity gradients.
pcl::PointCloud<pcl::IntensityGradient>::Ptr gradients(new pcl::PointCloud<pcl::IntensityGradient>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the RIFT descriptor for each point.
pcl::PointCloud<RIFT32>::Ptr descriptors(new pcl::PointCloud<RIFT32>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloudColor) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Convert the RGB to intensity.
pcl::PointCloudXYZRGBtoXYZI(*cloudColor, *cloudIntensity);

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZI, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudIntensity);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZI>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZI>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Compute the intensity gradients.
pcl::IntensityGradientEstimation < pcl::PointXYZI, pcl::Normal, pcl::IntensityGradient,
pcl::common::IntensityFieldAccessor<pcl::PointXYZI> > ge;
ge.setInputCloud(cloudIntensity);
ge.setInputNormals(normals);
ge.setRadiusSearch(0.03);
ge.compute(*gradients);

// RIFT estimation object.
pcl::RIFTEstimation<pcl::PointXYZI, pcl::IntensityGradient, RIFT32> rift;
rift.setInputCloud(cloudIntensity);
rift.setSearchMethod(kdtree);
// Set the intensity gradients to use.
rift.setInputGradient(gradients);
// Radius, to get all neighbors within.
rift.setRadiusSearch(0.02);
// Set the number of bins to use in the distance dimension.
rift.setNrDistanceBins(4);
// Set the number of bins to use in the gradient orientation dimension.
rift.setNrGradientBins(8);
// Note: you must change the output histogram size to reflect the previous values.

rift.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Intensity gradients, Radius, Distance bins, Gradient bins
* '''Output''': RIFT descriptors
* '''Publication''':
** [http://www.cs.illinois.edu/~slazebni/publications/pami05.pdf A Sparse Texture Representation Using Local Affine Regions] (Svetlana Lazebnik et al., 2005)
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_r_i_f_t_estimation.html pcl::RIFTEstimation]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_intensity_gradient_estimation.html pcl::IntensityGradientEstimation]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_intensity_gradient.html pcl::IntensityGradient]
* [http://robotica.unileon.es/~victorm/PCL_RIFT.tar.gz Download]
</div>

==NARF==

The Normal Aligned Radial Feature is the only descriptor here that does not take a point cloud as input. Instead, it works with range images. A range image is a common RGB image in which the distance to the point that corresponds to a certain pixel is encoded as a color value in the [https://en.wikipedia.org/wiki/Visible_spectrum visible light spectrum]: the points that are closer to the camera would be violet, while the points near the maximum sensor range would be red.

NARF also requires us to find suitable keypoints to compute the descriptor for. NARF keypoints are located near an object's corners, and this also requires to find the borders (transitions from foreground to background), which are trivial to find with a range image. Because of this lengthy pipeline, I will describe the whole process in different sections.

===Obtaining a range image===

Because we always work with point clouds, I will now explain how you can convert one into a range image, in order to use it for the NARF descriptor. PCL provides a couple of handy classes to perform the conversion, given that you fill the camera data correctly.

A range image can be created in two ways. First, we can use spherical projection, which would give us an image similar to the ones produced by a LIDAR sensor. Second, we can use planar projection, which is better suitable for camera-like sensors as the Kinect or the Xtion, and will not have the characteristic distortion of the first one.

====Spherical projection====

The following code will take a point cloud and create a range image from it, using spherical projection:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the range image object:

// Angular resolution is the angular distance between pixels.
// Kinect: 57° horizontal FOV, 43° vertical FOV, 640x480 (chosen here).
// Xtion: 58° horizontal FOV, 45° vertical FOV, 640x480.
float angularResolutionX = (float)(57.0f / 640.0f * (M_PI / 180.0f));
float angularResolutionY = (float)(43.0f / 480.0f * (M_PI / 180.0f));
// Maximum horizontal and vertical angles. For example, for a full panoramic scan,
// the first would be 360º. Choosing values that adjust to the real sensor will
// decrease the time it takes, but don't worry. If the values are bigger than
// the real ones, the image will be automatically cropped to discard empty zones.
float maxAngleX = (float)(60.0f * (M_PI / 180.0f));
float maxAngleY = (float)(50.0f * (M_PI / 180.0f));
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;
// Border size. If greater than 0, a border of "unobserved" points will be left
// in the image when it is cropped.
int borderSize = 1;

// Range image object.
pcl::RangeImage rangeImage;
rangeImage.createFromPointCloud(*cloud, angularResolutionX, angularResolutionY,
maxAngleX, maxAngleY, sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange, borderSize);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Range image");
viewer.showRangeImage(rangeImage);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

Here you can see an example of the output range image:

[[Image:Range_image_spherical.png|thumb|center|400px|Range image of a point cloud, using spherical projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Angular resolution, Maximum angle, Sensor pose, Coordinate frame, Noise level, Maximum range, Border size
* '''Output''': Range image (spherical projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image.html pcl::RangeImage]
* [http://robotica.unileon.es/~victorm/PCL_range_image_spherical.tar.gz Download]
</div>

====Planar projection====

As mentioned, planar projection will give better results with clouds taken from a depth camera:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Parameters needed by the planar range image object:

// Image size. Both Kinect and Xtion work at 640x480.
int imageSizeX = 640;
int imageSizeY = 480;
// Center of projection. here, we choose the middle of the image.
float centerX = 640.0f / 2.0f;
float centerY = 480.0f / 2.0f;
// Focal length. The value seen here has been taken from the original depth images.
// It is safe to use the same value vertically and horizontally.
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
// Sensor pose. Thankfully, the cloud includes the data.
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
// Noise level. If greater than 0, values of neighboring points will be averaged.
// This would set the search radius (e.g., 0.03 == 3cm).
float noiseLevel = 0.0f;
// Minimum range. If set, any point closer to the sensor than this will be ignored.
float minimumRange = 0.0f;

// Planar range image object.
pcl::RangeImagePlanar rangeImagePlanar;
rangeImagePlanar.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Visualize the image.
pcl::visualization::RangeImageVisualizer viewer("Planar range image");
viewer.showRangeImage(rangeImagePlanar);
while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_planar.png|thumb|center|400px|Range image of a point cloud, using planar projection.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Image size, Projection center, Focal length, Sensor pose, Coordinate frame, Noise level, Maximum range
* '''Output''': Range image (planar projection)
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/range_image_creation.php How to create a range image from a point cloud]
** [http://pointclouds.org/documentation/tutorials/range_image_visualization.php How to visualize a range image]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_planar.html pcl::RangeImagePlanar]
* [http://robotica.unileon.es/~victorm/PCL_range_image_planar.tar.gz Download]
</div>

If you prefer to do the conversion in real time while you inspect the cloud, PCL ships with an [https://github.com/PointCloudLibrary/pcl/tree/master/doc/tutorials/content/sources/openni_range_image_visualization example] that fetches an <span style="color:#FF1493">"openni_wrapper::DepthImage"</span> from an OpenNI device and creates the range image from it. You can adapt the code of the [[PCL/OpenNI_tutorial_1:_Installing_and_testing#Testing | example]] example from tutorial 1 to save it to disk with the function [http://docs.pointclouds.org/trunk/group__io.html#ga7291a029cdcde32ca3639d07dc6491b9 pcl::io::saveRangeImagePlanarFilePNG()].

===Extracting borders===

NARF keypoints are located near the edges of objects in the range image, so in order to find them, we first have to extract the borders. A border is defined as an abrupt change from foreground to background. In a range image, this can be easily seen because there is a "jump" in the depth value of two adjacent pixels.

[[Image:Range_image_border_detection.png|thumb|center|400px|Border detection on a range image (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

There are three types of borders. ''Object borders'' consist of the pixels (or points) located on the very edge of an object (the outermost points still belonging to the object). ''Shadow borders'' are points in the background on the edge of occlusions (empty areas in the background due to the objects in front covering them). Notice that, when the cloud is seen from the sensor's viewpoint, object and shadow points will seem adjacent. Finally, ''veil points'' are points interpolated between the previous two which appear in scans taken with LIDAR sensors, so we do not have to worry about them here.

<center><gallery widths=300px>
File:Range_image_border_type.png | Types of borders (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:Range_image_borders_example.png | Example of object and shadow borders on a cloud.
</gallery></center>

The algorithm basically compares a point's depth with the values of its neighbors, and if a big difference is found, we know it is due to a border. Points closer to the sensor will be marked as object borders, and the other ones as shadow borders.

PCL provides a class for extracting borders of a range image:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the borders.
pcl::PointCloud<pcl::BorderDescription>::Ptr borders(new pcl::PointCloud<pcl::BorderDescription>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Border extractor object.
pcl::RangeImageBorderExtractor borderExtractor(&rangeImage);

borderExtractor.compute(*borders);

// Visualize the borders.
pcl::visualization::RangeImageVisualizer* viewer = NULL;
viewer = pcl::visualization::RangeImageVisualizer::getRangeImageBordersWidget(rangeImage,
-std::numeric_limits<float>::infinity(),
std::numeric_limits<float>::infinity(),
false, *borders, "Borders");

while (!viewer->wasStopped())
{
viewer->spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_borders.png|thumb|center|400px|Borders found on the range image.]]

You can use the extractor's <span style="color:#FF1493">"getParameters()"</span> function to get a [http://docs.pointclouds.org/trunk/structpcl_1_1_range_image_border_extractor_1_1_parameters.html pcl::RangeImageBorderExtractor::Parameters] struct with the settings that will be used.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image
* '''Output''': Borders
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/range_image_border_extraction.php How to extract borders from range images]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_range_image_border_extractor.html pcl::RangeImageBorderExtractor]
* [http://robotica.unileon.es/~victorm/PCL_range_image_borders.tar.gz Download]
</div>

===Finding keypoints===

Citing the [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original publication]:''

''"We have the following requirements for our interest point extraction procedure:

# ''It must take information about borders and the surface structure into account.''
# ''It must select positions that can be reliably detected even if the object is observed from another perspective.''
# ''The points must be on positions that provide stable areas for normal estimation or the descriptor calculation in general."''

The procedure is the following: for every point in the range image, a score is computed that conveys how much the surface changes in its neighborhood (this is tuned with the ''support size'' σ, which is the diameter of the sphere used to find neighboring points). Also, the dominant direction of this change is computed. Then, this direction is compared with those of the neighbors, trying to find how stable the point is (if the directions are very different, that means the point is not stable, and that the surface around changes a lot). Points that are near the object's corners (but not exactly on the very edge) will be good keypoints, yet stable enough.

<center><gallery widths=300px>
File:NARF_keypoints.png | NARF keypoints (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
File:NARF_keypoints_support_sizes.png | Interest regions with a support size of 20cm (up) and 1m (down) (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).
</gallery></center>

In PCL, NARF keypoints can be found this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/visualization/range_image_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

pcl::RangeImageBorderExtractor borderExtractor;
// Keypoint detection object.
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
// The support size influences how big the surface of interest will be,
// when finding keypoints from the border information.
detector.getParameters().support_size = 0.2f;

detector.compute(*keypoints);

// Visualize the keypoints.
pcl::visualization::RangeImageVisualizer viewer("NARF keypoints");
viewer.showRangeImage(rangeImage);
for (size_t i = 0; i < keypoints->points.size(); ++i)
{
viewer.markPoint(keypoints->points[i] % rangeImage.width,
keypoints->points[i] / rangeImage.width,
// Set the color of the pixel to red (the background
// circle is already that color). All other parameters
// are left untouched, check the API for more options.
pcl::visualization::Vector3ub(1.0f, 0.0f, 0.0f));
}

while (!viewer.wasStopped())
{
viewer.spinOnce();
// Sleep 100ms to go easy on the CPU.
pcl_sleep(0.1);
}
}</syntaxhighlight>

[[Image:Range_image_NARF_keypoints.png|thumb|center|400px|NARF keypoints found on the range image.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Border extractor, Support Size
* '''Output''': NARF keypoints
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_keypoint_extraction.php How to extract NARF keypoint from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_keypoint.html pcl::NarfKeypoint]
* [http://robotica.unileon.es/~victorm/PCL_NARF_keypoints.tar.gz Download]
</div>

===Computing the descriptor===

We have created the range image from a point cloud, and we have extracted the borders in order to find good keypoints. Now it is time to compute the NARF descriptor for each keypoint.

The NARF descriptor encodes information about surface changes around a point. First, a local range patch is created around the point. It is like a small range image centered at that point, aligned with the normal (it would seem as if we were looking at the point along the normal). Then, a star pattern with ''n'' beams is overlaid onto the patch, also centered at the point. For every beam, a value is computed, that reflects how much the surface under it changes. The stronger the change is, and the closer to the center it is, the higher the final value will be. The ''n'' resulting values compose the final descriptor.

[[Image:NARF_descriptor.png|thumb|center|600px|Computing the NARF descriptor for a keypoint (image from [http://www.willowgarage.com/sites/default/files/icra2011_3dfeatures.pdf original paper]).]]

The descriptor right now is not invariant to rotations around the normal. To achieve this, the whole possible 360 degrees are binned into an histogram. The value of each bin is computed from the descriptor values according to the angle. Then, the bin with the highest value is considered the dominant orientation, and the descriptor is shifted according to it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/range_image/range_image_planar.h>
#include <pcl/features/range_image_border_extractor.h>
#include <pcl/keypoints/narf_keypoint.h>
#include <pcl/features/narf_descriptor.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the keypoints' indices.
pcl::PointCloud<int>::Ptr keypoints(new pcl::PointCloud<int>);
// Object for storing the NARF descriptors.
pcl::PointCloud<pcl::Narf36>::Ptr descriptors(new pcl::PointCloud<pcl::Narf36>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Convert the cloud to range image.
int imageSizeX = 640, imageSizeY = 480;
float centerX = (640.0f / 2.0f), centerY = (480.0f / 2.0f);
float focalLengthX = 525.0f, focalLengthY = focalLengthX;
Eigen::Affine3f sensorPose = Eigen::Affine3f(Eigen::Translation3f(cloud->sensor_origin_[0],
cloud->sensor_origin_[1],
cloud->sensor_origin_[2])) *
Eigen::Affine3f(cloud->sensor_orientation_);
float noiseLevel = 0.0f, minimumRange = 0.0f;
pcl::RangeImagePlanar rangeImage;
rangeImage.createFromPointCloudWithFixedSize(*cloud, imageSizeX, imageSizeY,
centerX, centerY, focalLengthX, focalLengthX,
sensorPose, pcl::RangeImage::CAMERA_FRAME,
noiseLevel, minimumRange);

// Extract the keypoints.
pcl::RangeImageBorderExtractor borderExtractor;
pcl::NarfKeypoint detector(&borderExtractor);
detector.setRangeImage(&rangeImage);
detector.getParameters().support_size = 0.2f;
detector.compute(*keypoints);

// The NARF estimator needs the indices in a vector, not a cloud.
std::vector<int> keypoints2;
keypoints2.resize(keypoints->points.size());
for (unsigned int i = 0; i < keypoints->size(); ++i)
keypoints2[i] = keypoints->points[i];
// NARF estimation object.
pcl::NarfDescriptor narf(&rangeImage, &keypoints2);
// Support size: choose the same value you used for keypoint extraction.
narf.getParameters().support_size = 0.2f;
// If true, the rotation invariant version of NARF will be used. The histogram
// will be shifted according to the dominant orientation to provide robustness to
// rotations around the normal.
narf.getParameters().rotation_invariant = true;

narf.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Range image, Key points, Support Size
* '''Output''': NARF descriptors
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/narf_feature_extraction.php How to extract NARF Features from a range image]
* '''Publication''':
** [http://europa.informatik.uni-freiburg.de/files/steder11icra.pdf Point Feature Extraction on 3D Range Scans Taking into Account Object Boundaries] (Bastian Steder et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_narf_descriptor.html pcl::NarfDescriptor]
* [http://robotica.unileon.es/~victorm/PCL_NARF_descriptor.tar.gz Download]
</div>

==RoPS==

The Rotational Projection Statistics (RoPS) feature is a bit different from the other descriptors because it works with a triangle mesh, so a previous triangulation step is needed for generating this mesh from the cloud. Apart from that, most concepts are similar.

In order to compute RoPS for a keypoint, the local surface is cropped according to a support radius, so only points and triangles lying inside are taken into account. Then, a local reference frame (LRF) is computed, giving the descriptor its rotational invariance. A coordinate system is created with the point as the origin, and the axes aligned with the LRF. Then, for every axis, several steps are performed.

First, the local surface is rotated around the current axis. The angle is determined by one of the parameters, which sets the number of rotations. Then, all points in the local surface are projected onto the XY, XZ and YZ planes. For each one, statistical information about the distribution of the projected points is computed, and concatenated to form the final descriptor.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/point_types_conversion.h>
#include <pcl/features/normal_3d.h>
#include <pcl/surface/gp3.h>
#include <pcl/features/rops_estimation.h>

// A handy typedef.
typedef pcl::Histogram<135> ROPS135;

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing both the points and the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr cloudNormals(new pcl::PointCloud<pcl::PointNormal>);
// Object for storing the ROPS descriptor for each point.
pcl::PointCloud<ROPS135>::Ptr descriptors(new pcl::PointCloud<ROPS135>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Perform triangulation.
pcl::concatenateFields(*cloud, *normals, *cloudNormals);
pcl::search::KdTree<pcl::PointNormal>::Ptr kdtree2(new pcl::search::KdTree<pcl::PointNormal>);
kdtree2->setInputCloud(cloudNormals);
pcl::GreedyProjectionTriangulation<pcl::PointNormal> triangulation;
pcl::PolygonMesh triangles;
triangulation.setSearchRadius(0.025);
triangulation.setMu(2.5);
triangulation.setMaximumNearestNeighbors(100);
triangulation.setMaximumSurfaceAngle(M_PI / 4); // 45 degrees.
triangulation.setNormalConsistency(false);
triangulation.setMinimumAngle(M_PI / 18); // 10 degrees.
triangulation.setMaximumAngle(2 * M_PI / 3); // 120 degrees.
triangulation.setInputCloud(cloudNormals);
triangulation.setSearchMethod(kdtree2);
triangulation.reconstruct(triangles);

// Note: you should only compute descriptors for chosen keypoints. It has
// been omitted here for simplicity.

// RoPs estimation object.
pcl::ROPSEstimation<pcl::PointXYZ, ROPS135> rops;
rops.setInputCloud(cloud);
rops.setSearchMethod(kdtree);
rops.setRadiusSearch(0.03);
rops.setTriangles(triangles.polygons);
// Number of partition bins that is used for distribution matrix calculation.
rops.setNumberOfPartitionBins(5);
// The greater the number of rotations is, the bigger the resulting descriptor.
// Make sure to change the histogram size accordingly.
rops.setNumberOfRotations(3);
// Support radius that is used to crop the local surface of the point.
rops.setSupportRadius(0.025);

rops.compute(*descriptors);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Triangles, Search method, Support radius, Number of rotations, Number of partition bins
* '''Output''': RoPS descriptors
* '''Publication''':
** [http://arxiv.org/pdf/1304.3192v1.pdf Rotational Projection Statistics for 3D Local Surface Description and Object Recognition] (Yulan Guo et al., 2013)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_r_o_p_s_estimation.html pcl::ROPSEstimation]
* [http://robotica.unileon.es/~victorm/PCL_RoPS.tar.gz Download]
</div>

=Global descriptors=

Global descriptors encode object geometry. They are not computed for individual points, but for a whole cluster that represents an object. Because of this, a preprocessing step ([[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Segmentation|segmentation]]) is required, in order to retrieve possible candidates.

Global descriptors are used for object recognition and classification, geometric analysis (object type, shape...), and pose estimation.

You should also know that many local descriptors can also be used as global ones. This can be done with descriptors that use a radius to search for neighbors (as PFH does). The trick is to compute it for one single point in the object cluster, and set the radius to the maximum possible distance between any two points (so all points in the cluster are considered as neighbors).

==VFH==

The Viewpoint Feature Histogram is based on the FPFH. Because the latter is invariant to the object's pose, the authors decided to expand it by including information about the viewpoint. Also, the FPFH is estimated once for the whole cluster, not for every point.

The VFH is made up by two parts: a viewpoint direction component, and an extended FPFH component. To compute the first one, the object's [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]] is found, which is the point that results from averaging the X, Y and Z coordinates of all points. Then, the vector between the viewpoint (the position of the sensor) and this centroid is computed, and normalized. Finally, for all points in the cluster, the angle between this vector and their normal is calculated, and the result is binned into an histogram. The vector is translated to each point when computing the angle because it makes the descriptor scale invariant.

The second component is computed like the FPFH (that results in 3 histograms for the 3 angular features, α, φ and θ), with some differences: it is only computed for the centroid, using the computed viewpoint direction vector as its normal (as the point, obviously, does not have a normal), and setting all the cluster's points as neighbors.

<center><gallery widths=300px>
File:VFH_viewpoint_component.png | Viewpoint component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
File:VFH_extended_FPFH_component.png | Extended FPFH component of the VFH (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).
</gallery></center>

The resulting 4 histograms (1 for the viewpoint component, 3 for the extended FPFH component) are concatenated to build the final VFH descriptor. By default, the bins are normalized using the total number of points in the cluster. This makes the VFH descriptor invariant to scale.

[[Image:VFH_histogram.png|thumb|center|400px| VFH histogram (image from [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf original paper]).]]

The PCL implementation computes an additional fifth histogram with the distances of the cluster points to the centroid (the Shape Distribution Component, SDC), increading the size of the output descriptor from 263 to 308. The SDC is taken from the CVFH descriptor that we will see in the next section, and makes the result more robust.

The VFH of an already clustered object can be computed this way:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the VFH descriptor.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// VFH estimation object.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
// Optionally, we can normalize the bins of the resulting histogram,
// using the total number of points.
vfh.setNormalizeBins(true);
// Also, we can normalize the SDC with the maximum size found between
// the centroid and any of the cluster's points.
vfh.setNormalizeDistance(false);

vfh.compute(*descriptor);
}</syntaxhighlight>

Because only one VFH descriptor is computed for the whole cluster, the size of the cloud object that stores the result will be 1.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, [Normalize bins], [Normalize SDC]
* '''Output''': VFH descriptor
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/vfh_estimation.php Estimating VFH signatures for a set of points]
* '''Publication''':
** [http://www.willowgarage.com/sites/default/files/Rusu10IROS.pdf Fast 3D Recognition and Pose Using the Viewpoint Feature Histogram] (Radu Bogdan Rusu et al., 2010)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_v_f_h_estimation.html pcl::VFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_VFH.tar.gz Download]
</div>

===CVFH===

The original VFH descriptor is not robust to occlusion or other sensor artifacts, or measurement errors. If the object cluster is missing many points, the resulting computed centroid will differ from the original, altering the final descriptor, and preventing a positive match from being found. Because of that, the Clustered Viewpoint Feature Histogram (CVFH) was introduced.

The idea is very simple: instead of computing a single VFH histogram for the whole cluster, the object is first divided in stable, smooth regions using [[PCL/OpenNI_tutorial_3:_Cloud_processing_(advanced)#Region_growing | region-growing segmentation]], that enforces several constraints in the distances and differences of normals of the points belonging to every region. Then, a VFH is computed for every region. Thanks to this, an object can be found in a scene, as long as at least one of its regions is fully visible.

<center><gallery widths=300px>
File:CVFH_occlusion.png | Typical occlusion issues in a point cloud (image from original paper).
File:CVFH_regions.png | Object regions computed for the CVFH (image from original paper).
</gallery></center>

Additionally, a Shape Distribution Component (SDC) is also computed and included. It encodes information about the distribution of the points arond the region's centroid, measuring the distances. The SDC allows to differentiate objects with similar characteristics (size and normal distribution), like two planar surfaces from each other.

The authors proposed to discard the histogram normalization step that is performed in VFH. This has the effect of making the descriptor dependant of scale, so an object of a certain size would not match a bigger or smaller copy of itself. It also makes CVFH more robust to occlusion.

CVFH is invariant to the camera roll angle, like most global descriptors. This is so because rotations about that camera axis do not change the observable geometry that descriptors are computed from, limiting the pose estimation to 5 DoF. The use of a Camera Roll Histogram (CRH) has been proposed to overcome this.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// CVFH estimation object.
pcl::CVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> cvfh;
cvfh.setInputCloud(object);
cvfh.setInputNormals(normals);
cvfh.setSearchMethod(kdtree);
// Set the maximum allowable deviation of the normals,
// for the region segmentation step.
cvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
// Set the curvature threshold (maximum disparity between curvatures),
// for the region segmentation step.
cvfh.setCurvatureThreshold(1.0);
// Set to true to normalize the bins of the resulting histogram,
// using the total number of points. Note: enabling it will make CVFH
// invariant to scale just like VFH, but the authors encourage the opposite.
cvfh.setNormalizeBins(false);

cvfh.compute(*descriptors);
}</syntaxhighlight>

You can further customize the segmentation step with <span style="color:#FF1493">"setClusterTolerance()"</span> (to set the maximum Euclidean distance between points in the same cluster) and <span style="color:#FF1493">"setMinPoints()"</span>. The size of the output will be equal to the number of regions the object was divided in. Also, check the functions <span style="color:#FF1493">"getCentroidClusters()"</span> and <span style="color:#FF1493">"getCentroidNormalClusters()"</span>, you can use them to get information about the centroids used to compute the different CVFH descriptors.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points]
* '''Output''': CVFH descriptors
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6130296 CAD-Model Recognition and 6DOF Pose Estimation Using 3D Cues] (requires IEEE Xplore subscription) (Aitor Aldoma et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_c_v_f_h_estimation.html pcl::CVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_CVFH.tar.gz Download]
</div>

====OUR-CVFH====

The Oriented, Unique and Repeatable CVFH expands the previous descriptor, adding the computation of an unique reference frame to make it more robust.

OUR-CVFH relies on the use of Semi-Global Unique Reference Frames (SGURFs), which are repeatable coordinate systems computed for each region. Not only they remove the invariance to camera roll and allow to extract the 6DoF pose directly without additional steps, but they also improve the spatial descriptiveness.

The first part of the computation is akin to CVFH, but after segmentation, the points in each region are filtered once more according to the difference between their normals and the region's average normal. This results in better shaped regions, improving the estimation of the Reference Frames (RFs).

After this, the SGURF is computed for each region. Disambiguation is performed to decide the sign of the axes, according to the points' distribution. If this is not enough and the sign remains ambiguous, multiple RFs will need to be created to account for it. Finally, the OUR-CVFH descriptor is computed. The original Shape Distribution Component (SDC) is discarded, and the surface is now described according to the RFs.

[[Image:OUR-CVFH.png|thumb|center|600px| SGURF frame and resulting histogram of a region (image from [http://vision.deis.unibo.it/fede/papers/dagm12.pdf original paper]).]]

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/our_cvfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the OUR-CVFH descriptors.
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptors(new pcl::PointCloud<pcl::VFHSignature308>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// OUR-CVFH estimation object.
pcl::OURCVFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> ourcvfh;
ourcvfh.setInputCloud(object);
ourcvfh.setInputNormals(normals);
ourcvfh.setSearchMethod(kdtree);
ourcvfh.setEPSAngleThreshold(5.0 / 180.0 * M_PI); // 5 degrees.
ourcvfh.setCurvatureThreshold(1.0);
ourcvfh.setNormalizeBins(false);
// Set the minimum axis ratio between the SGURF axes. At the disambiguation phase,
// this will decide if additional Reference Frames need to be created, if ambiguous.
ourcvfh.setAxisRatio(0.8);

ourcvfh.compute(*descriptors);
}</syntaxhighlight>

You can use the <span style="color:#FF1493">"getTransforms()"</span> function to get the transformations aligning the cloud to the corresponding SGURF.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Angle threshold, Curvature threshold, [Normalize bins], [Cluster tolerance], [Minimum points], [Axis ratio]
* '''Output''': OUR-CVFH descriptors
* '''Publication''':
** [http://vision.deis.unibo.it/fede/papers/dagm12.pdf OUR-CVFH – Oriented, Unique and Repeatable Clustered Viewpoint Feature Histogram for Object Recognition and 6DOF Pose Estimation] (Aitor Aldoma et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_o_u_r_c_v_f_h_estimation.html pcl::OURCVFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_OUR-CVFH.tar.gz Download]
</div>

==ESF==

The Ensemble of Shape Functions (ESF) is a combination of 3 different shape functions that describe certain properties of the cloud's points: distances, angles and area. This descriptor is very unique because it does not require normal information. Actually, it does not need any preprocessing, as it is robust to noise and incomplete surfaces.

The algorithm uses a voxel grid as an approximation of the real surface. It iterates through all the points in the cloud: for every iteration, 3 random points are chosen. For these points, the shape functions are computed:

* '''D2''': this function computes the distances between point pairs (3 overall). Then, for every pair, it checks if the line that connects both points lies entirely inside the surface, entirely outside (crossing free space), or both. Depending on this, the distance value will be binned to one of three possible histograms: IN, OUT or MIXED.
* '''D2 ratio''': an additional histogram for the ratio between parts of the line inside the surface, and parts outside. This value will be 0 if the line is completely outside, 1 if completely inside, and some value in between if mixed.
* '''D3''': this computes the square root of the area of the triangle formed by the 3 points. Like D2, the result is also classified as IN, OUT or MIXED, each with its own histogram.
* '''A3''': this function computes the angle formed by the points. Then, the value is binned depending on how the line opposite to the angle is (once again, as IN, OUT or MIXED).

[[Image:ESF.png|thumb|center|600px| ESF descriptor (image from [http://www.inf.ethz.ch/personal/zeislb/publications/aldoma_2012jram_PCLTutorial.pdf this paper]).]]

After the loop is over, we are left with 10 subhistograms (IN, OUT and MIXED for D2, D3 and A3, and an additional one for the ratio). Each one has 64 bins, so the size of the final ESF descriptor is 640.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/esf.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the ESF descriptor.
pcl::PointCloud<pcl::ESFSignature640>::Ptr descriptor(new pcl::PointCloud<pcl::ESFSignature640>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// ESF estimation object.
pcl::ESFEstimation<pcl::PointXYZ, pcl::ESFSignature640> esf;
esf.setInputCloud(object);

esf.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster)
* '''Output''': ESF descriptor
* '''Publication''':
** [http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6181760 Ensemble of shape functions for 3D object classification] (requires IEEE Xplore subscription) (Walter Wohlkinger and Markus Vincze, 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_e_s_f_estimation.html pcl::ESFEstimation]
* [http://robotica.unileon.es/~victorm/PCL_ESF.tar.gz Download]
</div>

==GFPFH==

As you may have guessed, GFPFH stands for Global Fast Point Feature Histogram, the global version of the FPFH descriptor. GFPFH was designed for the task of helping a robot navigate its environment, having some context of the objects around it.

The first step before being able to compute the descriptor is surface categorization. A set of logical primitives (the classes, or categories) is created, which depends on the type of objects we expect the robot to find on the scene. For example, if we know there will be a coffee mug, we create three: one for the handle, and the other two for the outer and inner faces. Then, FPFH descriptors are computed, and everything is fed to a [https://en.wikipedia.org/wiki/Conditional_random_field Conditional Random Field] (CRF) algorithm. The CRF will label each surface with one of the previous categories, so we end up with a cloud where each point has been classified depending of the type of object (or object's region) it belongs to.

[[Image:FPFH_CRF.png|thumb|center|300px| Classification of objects made with FPFH and CRF (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

Now, the GFPFH descriptor can be computed with the result of the classification step. It will encode what the object is made of, so the robot can easily recognize it. First, an [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Octree | octree]] is created, dividing the object in voxel leaves. For every leaf, a set of probabilities is created, one for each class. Each one stores the probability of that leaf belonging to the class, and it is computed according to the number of points in that leaf that have been labelled as that class, and the total number of points. Then, for every pair of leaves in the octree, a line is casted, connecting them. Every leaf in its path is checked for occupancy, storing the result in an histogram. If the leaf is empty (free space), a value of 0 is saved. Otherwise, the leaf probabilities are used.

[[Image:GFPFH.png|thumb|center|500px| Computing the GFPFH with a voxel grid (image from [https://www.willowgarage.com/sites/default/files/iccv09.pdf original paper]).]]

The following code will compute the GFPFH for a cloud with label information. The categorization step is up to you, as it depends largely on the type of the scene, and the use you are going to give it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/gfpfh.h>

int
main(int argc, char** argv)
{
// Cloud for storing the object.
pcl::PointCloud<pcl::PointXYZL>::Ptr object(new pcl::PointCloud<pcl::PointXYZL>);
// Object for storing the GFPFH descriptor.
pcl::PointCloud<pcl::GFPFHSignature16>::Ptr descriptor(new pcl::PointCloud<pcl::GFPFHSignature16>);

// Note: you should have performed preprocessing to cluster out the object
// from the cloud, and save it to this individual file.

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZL>(argv[1], *object) != 0)
{
return -1;
}

// Note: you should now perform classification on the cloud's points. See the
// original paper for more details. For this example, we will now consider 4
// different classes, and randomly label each point as one of them.
for (size_t i = 0; i < object->points.size(); ++i)
{
object->points[i].label = 1 + i % 4;
}

// ESF estimation object.
pcl::GFPFHEstimation<pcl::PointXYZL, pcl::PointXYZL, pcl::GFPFHSignature16> gfpfh;
gfpfh.setInputCloud(object);
// Set the object that contains the labels for each point. Thanks to the
// PointXYZL type, we can use the same object we store the cloud in.
gfpfh.setInputLabels(object);
// Set the size of the octree leaves to 1cm (cubic).
gfpfh.setOctreeLeafSize(0.01);
// Set the number of classes the cloud has been labelled with (default is 16).
gfpfh.setNumberOfClasses(4);

gfpfh.compute(*descriptor);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Labels, Number of classes, Leaf size
* '''Output''': GFPFH descriptor
* '''Publication''':
** [https://www.willowgarage.com/sites/default/files/iccv09.pdf Detecting and Segmenting Objects for Mobile Manipulation] (Radu Bogdan Rusu et al., 2009)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_f_p_f_h_estimation.html pcl::GFPFHEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GFPFH.tar.gz Download]
</div>

==GRSD==

The global version of the Radius-based Surface Descriptor works in a similar fashion to GFPFH. A voxelization and a surface categorization step are performed beforehand, labelling all surface patches according to the geometric category (plane, cylinder, edge, rim, sphere), using RSD. Then, the whole cluster is classified into one of these categories, and the GRSD descriptor is computed from this.

[[Image:GRSD.png|thumb|center|400px| Classification of objects for GRSD and resulting histogram (image from [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf original paper]).]]

To compute it:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/grsd.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing the GRSD descriptors for each point.
pcl::PointCloud<pcl::GRSDSignature21>::Ptr descriptors(new pcl::PointCloud<pcl::GRSDSignature21>());

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Note: you would usually perform downsampling now. It has been omitted here
// for simplicity, but be aware that computation can take a long time.

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// GRSD estimation object.
GRSDEstimation<PointXYZ, Normal, GRSDSignature21> grsd;
grsd.setInputCloud(cloud);
grsd.setInputNormals(normals);
grsd.setSearchMethod(kdtree);
// Search radius, to look for neighbors. Note: the value given here has to be
// larger than the radius used to estimate the normals.
grsd.setRadiusSearch(0.05);

grsd.compute(*descriptors);
}</syntaxhighlight>

<span style="color:#606060">'''''NOTE: This code will only compile with PCL versions 1.8 and above (the current trunk).'''''</span>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points (cluster), Normals, Search method, Radius
* '''Output''': GRSD descriptor
* '''Publications''':
** [https://ias.in.tum.de/_media/spezial/bib/grsd10humanoids.pdf Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation] (Zoltan-Csaba Marton et al., 2010)
** [https://ias.cs.tum.edu/_media/spezial/bib/marton11ijrr.pdf Combined 2D-3D Categorization and Classification for Multimodal Perception Systems] (Zoltan-Csaba Marton et al., 2011)
** [http://www.mi.t.u-tokyo.ac.jp/top/downloadpublication/36 Voxelized Shape and Color Histograms for RGB-D] (Zoltan-Csaba Marton et al., 2011)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_g_r_s_d_estimation.html pcl::GRSDEstimation]
* [http://robotica.unileon.es/~victorm/PCL_GRSD.tar.gz Download]
</div>

=Saving and loading=

You can save a descriptor to a file just [[PCL/OpenNI_tutorial_2:_Cloud_processing_(basic)#Writing_to_file|like with any other cloud type]]. One caveat, though. If you are using a descriptor that has its own custom type, like <span style="color:#FF1493">"PFHSignature125"</span>, everything will be OK. But with descriptors that do not (where you have to use <span style="color:#FF1493">"pcl::Histogram<>"</span>), you will get this error: <span style="color:#FF1493">''"POINT_TYPE_NOT_PROPERLY_REGISTERED"''</span>. In order to save or load from a file, PCL's IO functions need to know about the number, type and size of the fields. To solve this, you will have to properly register a new point type for your descriptor. For example, this will work for the [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#RoPS|RoPS]] descriptor example we saw earlier:

<syntaxhighlight lang=CPP>POINT_CLOUD_REGISTER_POINT_STRUCT(ROPS135,
(float[135], histogram, histogram)
)</syntaxhighlight>

Add the previous to your code (change it accordingly), and you will be able to save and load descriptors as usual.

=Visualization=

Sometimes it is desired to check a visual representation of a descriptor, perhaps to analyze the distribution of data over different bins. Because they are saved as histograms, this is something trivial to do. PCL offers a couple of classes to do this.

==PCLHistogramVisualizer==

<span style="color:#FF1493">"PCLHistogramVisualizer"</span> is the simplest way to plot an histogram. The class has little functionality, but it does its job. Only one call is necessary to give the histogram and its size:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/histogram_visualizer.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLHistogramVisualizer viewer;
// We need to set the size of the descriptor beforehand.
viewer.addFeatureHistogram(*descriptor, 308);

viewer.spin();
}</syntaxhighlight>

[[Image:Histogram_visualizer_VFH.png|thumb|center|500px| VFH histogram seen with the Histogram Visualizer.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_histogram_visualizer.html pcl::visualization::PCLHistogramVisualizer]
* [http://robotica.unileon.es/~victorm/PCL_histogram_visualizer.tar.gz Download]
</div>

==PCLPlotter==

This class has all the methods from <span style="color:#FF1493">"PCLHistogramVisualizer"</span> (which will be deprecated soon) plus a lot more features. The code is almost the same:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/features/vfh.h>
#include<pcl/visualization/pcl_plotter.h>

int
main(int argc, char** argv)
{
// Clouds for storing everything.
pcl::PointCloud<pcl::PointXYZ>::Ptr object(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::VFHSignature308>::Ptr descriptor(new pcl::PointCloud<pcl::VFHSignature308>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *object) != 0)
{
return -1;
}

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(object);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Estimate VFH descriptor.
pcl::VFHEstimation<pcl::PointXYZ, pcl::Normal, pcl::VFHSignature308> vfh;
vfh.setInputCloud(object);
vfh.setInputNormals(normals);
vfh.setSearchMethod(kdtree);
vfh.setNormalizeBins(true);
vfh.setNormalizeDistance(false);
vfh.compute(*descriptor);

// Plotter object.
pcl::visualization::PCLPlotter plotter;
// We need to set the size of the descriptor beforehand.
plotter.addFeatureHistogram(*descriptor, 308);

plotter.plot();
}</syntaxhighlight>

[[Image:PCL_plotter_VFH.png|thumb|center|500px| VFH histogram seen with the PCLPlotter class.]]

If you have raw data (such as a vector of floats) you can use the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html#aaf23b2b1c2f91c517cfde387ee1b654e addHistogramData()] function to plot it as an histogram.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Descriptor, descriptor size, [Window size], [Background color], [Y axis range]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/pcl_plotter.php PCLPlotter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_plotter.html pcl::visualization::PCLPlotter]
* [http://robotica.unileon.es/~victorm/PCL_plotter.tar.gz Download]
</div>

==PCL Viewer==

This program, included with PCL, will also let you open and visualize a saved descriptor. Internally, it uses [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29#PCLPlotter|PCLPlotter]]. You can invoke the viewer from the command line like this, so it comes handy:

<syntaxhighlight lang=Bash>pcl_viewer <descriptor_file></syntaxhighlight>

[[Image:PCL_viewer_VFH.png|thumb|center|500px| VFH histogram seen with ''pcl_viewer''.]]

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 3: Cloud processing (advanced)

2015-11-01T23:09:09Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

Most of the techniques seen in the [[PCL/OpenNI tutorial 2: Cloud processing (basic) | previous tutorial]] focused on ''preprocessing'', that is, performing certain operations on the cloud to get it ready for further analysis or work. Downsampling, removing outliers, surface smoothing, estimating the normals...

This new tutorial will teach you many interesting things that can be done with point clouds after the preparation step, such as registration, segmentation and model matching, and other advanced operations.

=Registration=

''Registration'' is the technique of aligning two point clouds, like pieces of a puzzle. To be precise, the algorithm finds a set of correspondences between them, which would mean that there is an area of the scene that has been captured in both clouds. A ''linear transformation'' is then computed, which outputs a matrix that contains a rotation and a translation. These are the operations that you would apply to one of the clouds so it would get in place with respect to the other, with the intersecting areas overlapping.

The best results are achieved with clouds that are very similar, so you should try to minimize the transformations between them. Meaning, do not run like crazy with a Kinect in your hands, grabbing frames, and expect PCL to match the clouds. It is better to move the sensor gingerly and at steady intervals. If you use a robotic arm or a rotating tabletop with precise angles, even better.

Registration is a very useful technique because it lets you retrieve full, complete and continous [http://pointclouds.org/documentation/tutorials/in_hand_scanner.php models] of a scene or object. It is also expensive for the CPU. Optimizations have been developed that allow to do cloud matching in real time with the GPU, like [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php KinFu] does.

[[Image:Pairwise_registration.jpg|thumb|center|650px|Flowchart diagram showing an iteration of the registration algorithm (image from http://pointclouds.org/).]]

The previous diagram illustrates how the procedure is done. There are mainly two different methods to choose from:

* '''ICP registration''': ICP stands for [https://en.wikipedia.org/wiki/Iterative_closest_point Iterative Closest Point]. It is an algorithm that will find the best transformation that minimizes the distance from the source point cloud to the target one. The problem is that it will do it by associating every point of the source cloud to its "twin" in the target cloud in a linear way, so it can be considered a brute force method. It the clouds are too big, the algorithm will take its time to finish, so try downsampling them first.
* '''Feature-based registration''': the algorithm finds a set of keypoints in each cloud, computes a local descriptor for each (we explain what local descriptors are in [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|the next tutorial]]; they are like individual signatures of points), and then performs a search to see if the clouds have keypoints in common. If at least 3 correspondences are found, a transformation can be computed. For accurate results, several correspondences must be found. This method is (or should be) faster than the first, because matching is only done for the keypoints, not the whole cloud.

ICP will probably fail if the difference between the clouds is too big. Usually, you will use features first to perform an initial rough alignment of the clouds, and then use ICP to refine it with precision. Feature-based registration is basically identical to the [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29|3D object recognition problem]], so we will not look into it here. Features, descriptors and recognition will be dealt with in the next tutorials.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/registration_api.php The PCL Registration API]
** [http://www.pointclouds.org/assets/icra2013/Miller_presentation_ICRA13.pdf Clean Models from Noisy Sensors: Registration and Reconstruction Techniques]
** [http://www.pointclouds.org/assets/icra2013/registration.pdf 3D Registration with PCL]
** [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Registration.pdf Point Cloud Registration]
</div>

==Iterative Closest Point (ICP)==

PCL offers several implementations of the ICP algorithm. We will see the general, unoptimized one, but you may want to try one of the optimized variants listed below in the API section. The code for aligning (registering) two point clouds is the following:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/registration/icp.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr sourceCloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr targetCloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr finalCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *sourceCloud) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *targetCloud) != 0)
{
return -1;
}

// ICP object.
pcl::IterativeClosestPoint<pcl::PointXYZ, pcl::PointXYZ> registration;
registration.setInputSource(sourceCloud);
registration.setInputTarget(targetCloud);

registration.align(*finalCloud);
if (registration.hasConverged())
{
std::cout << "ICP converged." << std::endl
<< "The score is " << registration.getFitnessScore() << std::endl;
std::cout << "Transformation matrix:" << std::endl;
std::cout << registration.getFinalTransformation() << std::endl;
}
else std::cout << "ICP did not converge." << std::endl;
}</syntaxhighlight>

The cloud saved in <span style="color:#FF1493">"finalCloud"</span> is the same as <span style="color:#FF1493">"sourceCloud"</span> (it has the same points), but the transformation found by ICP has been applied, so when visualized next to <span style="color:#FF1493">"targetCloud"</span> they should match properly (errors are inevitable and depend of many factors, like the quality of the sensor, the distance between the clouds, the algorithm used...).

<center><gallery widths=300px>
File:Registration_cloud1.png | Source point cloud.
File:Registration_cloud2.png | Target point cloud.
File:Registration_before.png | Both clouds shown together before registration.
File:Registration_after.png | Both clouds shown together after registration.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Source cloud, Target cloud
* '''Output''': Registered source cloud, Transformation matrix
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/iterative_closest_point.php How to use iterative closest point]
** [http://pointclouds.org/documentation/tutorials/pairwise_incremental_registration.php How to incrementally register pairs of clouds]
** [http://pointclouds.org/documentation/tutorials/interactive_icp.php Interactive Iterative Closest Point]
** [http://pointclouds.org/documentation/tutorials/normal_distributions_transform.php How to use Normal Distributions Transform]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_iterative_closest_point.html pcl::IterativeClosestPoint]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_iterative_closest_point_with_normals.html pcl::IterativeClosestPointWithNormals]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_iterative_closest_point_non_linear.html pcl::IterativeClosestPointNonLinear]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_joint_iterative_closest_point.html pcl::JointIterativeClosestPoint]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_distributions_transform.html pcl::NormalDistributionsTransform]
* [http://robotica.unileon.es/~victorm/PCL_registration.tar.gz Download]
</div>

=Model fitting (RANSAC)=

Given a set of data, it is possible to determine if a part of it fits a certain mathematical model, using an iterative method known as [https://en.wikipedia.org/wiki/RANSAC RANSAC (Random Sample Consensus)]. RANSAC works under the assumption that the data contains ''inliers'' (data can can be adjusted to the model, even with a little noise) and ''outliers'' (data that does not fit the model at all). The algoritm is non-deterministic: every iteration, the accuracy of the result improves. To be precise, the probability of the result being the correct one increases, though it will never be of 100%.

<center><gallery widths=300px>
File:RANSAC_before.png | Set of data points containing outliers and inliers fitting a line model (image from Wikimedia Commons).
File:RANSAC_after.png | The same set, after applying RANSAC to find the parameters of the line model, discarding outliers (image from Wikimedia Commons).
</gallery></center>

PCL offers implementations of RANSAC to work with point clouds. For example, you can find which points in the cloud fit the model of a sphere, and the procedure will return the parameters of said model. This is how you would do it:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/ransac.h>
#include <pcl/sample_consensus/sac_model_sphere.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr inlierPoints(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// RANSAC objects: model and algorithm.
pcl::SampleConsensusModelSphere<pcl::PointXYZ>::Ptr model(new pcl::SampleConsensusModelSphere<pcl::PointXYZ>(cloud));
pcl::RandomSampleConsensus<pcl::PointXYZ> ransac(model);
// Set the maximum allowed distance to the model.
ransac.setDistanceThreshold(0.01);
ransac.computeModel();

std::vector<int> inlierIndices;
ransac.getInliers(inlierIndices);

// Copy all inliers of the model to another cloud.
pcl::copyPointCloud<pcl::PointXYZ>(*cloud, inlierIndices, *inlierPoints);
}</syntaxhighlight>

There are many more models to choose from. The most important ones are:

* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_circle2_d.html pcl::SampleConsensusModelCircle2D]: A 2D circle on the X-Y plane.
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_circle3_d.html pcl::SampleConsensusModelCircle3D]: A 3D circle (any plane).
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_cone.html pcl::SampleConsensusModelCone]: A cone.
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_cylinder.html pcl::SampleConsensusModelCylinder]: A cylinder.
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_line.html pcl::SampleConsensusModelLine]: A line.
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_plane.html pcl::SampleConsensusModelPlane]: A plane.
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_sphere.html pcl::SampleConsensusModelSphere]: A sphere.
* [http://docs.pointclouds.org/trunk/classpcl_1_1_sample_consensus_model_stick.html pcl::SampleConsensusModelStick]: A stick (a line with user-given minimum/maximum width).

Also, the RANSAC class offers some functions to configure its behavior. For example, with <span style="color:#FF1493">"setMaxIterations()"</span> you can specify the maximum number of iterations the algoritm will run for, instead of using a distance threshold (if you do not set a stop condition, RANSAC will just run forever, getting a slightly better result each time).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Model, Model threshold
* '''Output''': Vector of Point indices (inliers)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/random_sample_consensus.php How to use Random Sample Consensus model]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_random_sample_consensus.html pcl::RandomSampleConsensus]
* [http://robotica.unileon.es/~victorm/PCL_sphere_model.tar.gz Download]
</div>

==Projecting points==

After you have recovered the coefficients of a model, it is possible to project the points of a cloud onto said model. For example, if the model is a plane, after projection the points of the cloud would all lie on the plane.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/filters/project_inliers.h>
#include <pcl/visualization/cloud_viewer.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudNoPlane(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr planePoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr projectedPoints(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Get the plane model, if present.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setOptimizeCoefficients(true);
pcl::PointIndices::Ptr inlierIndices(new pcl::PointIndices);
segmentation.segment(*inlierIndices, *coefficients);

if (inlierIndices->indices.size() == 0)
std::cout << "Could not find a plane in the scene." << std::endl;
else
{
std::cerr << "Plane coefficients: " << coefficients->values[0] << " "
<< coefficients->values[1] << " "
<< coefficients->values[2] << " "
<< coefficients->values[3] << std::endl;

// Create a second point cloud that does not have the plane points.
// Also, extract the plane points to visualize them later.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloud);
extract.setIndices(inlierIndices);
extract.filter(*planePoints);
extract.setNegative(true);
extract.filter(*cloudNoPlane);

// Object for projecting points onto a model.
pcl::ProjectInliers<pcl::PointXYZ> projection;
// We have to specify what model we used.
projection.setModelType(pcl::SACMODEL_PLANE);
projection.setInputCloud(cloudNoPlane);
// And we have to give the coefficients we got.
projection.setModelCoefficients(coefficients);
projection.filter(*projectedPoints);

// Visualize everything.
pcl::visualization::CloudViewer viewerPlane("Plane");
viewerPlane.showCloud(planePoints);
while (!viewerPlane.wasStopped())
{
// Do nothing but wait.
}
pcl::visualization::CloudViewer viewerProjection("Projected points");
viewerProjection.showCloud(projectedPoints);
while (!viewerProjection.wasStopped())
{
// Do nothing but wait.
}
}
}</syntaxhighlight>

After running the example, you will see that all points have been "flattened" onto the plane (if there was one in the scene, that is). The program will first show you the points that have been fitted to a plane. When you quit with <span style="color:#228B22">'''Alt+F4'''</span> or <span style="color:#228B22">'''Q'''</span>, a second window will open displaying the results of the projection.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Model type, Model coefficients
* '''Output''': Projected points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/project_inliers.php Projecting points using a parametric model]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_project_inliers.html pcl::ProjectInliers]
* [http://robotica.unileon.es/~victorm/PCL_point_projection.tar.gz Download]
</div>

=Segmentation=

''Segmentation'' consists of breaking the cloud apart in different pieces or sections: groups of points called ''clusters'' (which is why it is also referred to as ''clustering''). The idea is to divide it in several parts to be processed independently. Ideally, every cluster would belong to the logical notion of "object". For example, for a cloud that shows 3 boxes on a table, 4 clusters would be created: one for the table, and one for each of the boxes.

There are many segmentation techniques, each one with its own criteria for grouping points together. Some consider the distance between points, some take into account the normals, or even the texture, too. Here I will give you an overview of each one.

Segmentation, together with all other methods seen so far, will allow you to retrieve the models of individual objects from a scene. Also, it is possible to segment certain shapes like planes or cylinders. If you need example point clouds to practice your segmentation skills with, take a look at the [http://www.acin.tuwien.ac.at/?id=289 OSD] (Object Segmentation Database).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://www.pointclouds.org/assets/icra2013/pcl_organized_segmentation.pdf Organized Point Cloud Segmentation]
</div>

==Euclidean==

Euclidean segmentation is the simplest of all. It checks the distance between two points. If it is less than a threshold, both are considered to belong in the same cluster. It works like a [https://en.wikipedia.org/wiki/Flood_fill flood fill] algorithm: a point in the cloud is "marked" as "chosen" for the cluster. Then, it spreads like a virus to all other points that are near enough, and from those to even more points, until none new can be added. Then, a new cluster is initialized, and the procedure starts again with the remaining unmarked points.

In PCL, Euclidean segmentation is done as follows:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/segmentation/extract_clusters.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object for searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
kdtree->setInputCloud(cloud);

// Euclidean clustering object.
pcl::EuclideanClusterExtraction<pcl::PointXYZ> clustering;
// Set cluster tolerance to 2cm (small values may cause objects to be divided
// in several clusters, whereas big values may join objects in a same cluster).
clustering.setClusterTolerance(0.02);
// Set the minimum and maximum number of points that a cluster can have.
clustering.setMinClusterSize(100);
clustering.setMaxClusterSize(25000);
clustering.setSearchMethod(kdtree);
clustering.setInputCloud(cloud);
std::vector<pcl::PointIndices> clusters;
clustering.extract(clusters);

// For every cluster...
int currentClusterNum = 1;
for (std::vector<pcl::PointIndices>::const_iterator i = clusters.begin(); i != clusters.end(); ++i)
{
// ...add all its points to a new cloud...
pcl::PointCloud<pcl::PointXYZ>::Ptr cluster(new pcl::PointCloud<pcl::PointXYZ>);
for (std::vector<int>::const_iterator point = i->indices.begin(); point != i->indices.end(); point++)
cluster->points.push_back(cloud->points[*point]);
cluster->width = cluster->points.size();
cluster->height = 1;
cluster->is_dense = true;

// ...and save it to disk.
if (cluster->points.size() <= 0)
break;
std::cout << "Cluster " << currentClusterNum << " has " << cluster->points.size() << " points." << std::endl;
std::string fileName = "cluster" + boost::to_string(currentClusterNum) + ".pcd";
pcl::io::savePCDFileASCII(fileName, *cluster);

currentClusterNum++;
}
}</syntaxhighlight>

You may have to play with the cluster tolerance parameter until you get good results. Also, it is recommended to perform planar segmentation (we will see it later) to remove the floor and/or table(s) from the scene, or the results may be wrong.

After the program finishes, you can visualize all clusters at the same time with:

<syntaxhighlight lang=Bash>pcl_viewer cluster*</syntaxhighlight>

<center><gallery widths=300px>
File:EuclideanClusteringBefore.png | Original cloud.
File:EuclideanClusteringAfter.png | Clusters recovered after Euclidean segmentation (each cluster is renderer with a different color).
</gallery></center>

All segmentation objects provide a <span style="color:#FF1493">"setIndices()"</span> function. You can use it to provide the indices of the points you want to apply clustering to, instead of the whole cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Cluster tolerance, Minimum cluster size, Maximum cluster size, Search method, [Indices]
* '''Output''': Vector of Point indices (clusters)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/cluster_extraction.php Euclidean Cluster Extraction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_euclidean_cluster_extraction.html pcl::EuclideanClusterExtraction]
* [http://robotica.unileon.es/~victorm/PCL_euclidean_clustering.tar.gz Download]
</div>

===Conditional===

Conditional Euclidean segmentation works the same way as the standard one seen above, with one exception. Apart from the distance check, points need also to meet a special, custom requirement for them to be added to a cluster. This condition is user-specified. It boils down to this: for every pair of points (the first one, the ''seed'', is being processed in this moment, the second one, ''candidate'', is a neighbor of the former that is being tested) a custom function will be called. This function has a certain signature: it receives
# a copy of the two points so we can perform our own tests, and
# the squared distance
and returns a boolean value. If the value is true, the candidate may be added to the cluster. If false, it will not, even if it passed the distance check.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/segmentation/conditional_euclidean_clustering.h>

#include <iostream>

// If this function returns true, the candidate point will be added
// to the cluster of the seed point.
bool
customCondition(const pcl::PointXYZ& seedPoint, const pcl::PointXYZ& candidatePoint, float squaredDistance)
{
// Do whatever you want here.
if (candidatePoint.y < seedPoint.y)
return false;

return true;
}

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Conditional Euclidean clustering object.
pcl::ConditionalEuclideanClustering<pcl::PointXYZ> clustering;
clustering.setClusterTolerance(0.02);
clustering.setMinClusterSize(100);
clustering.setMaxClusterSize(25000);
clustering.setInputCloud(cloud);
// Set the function that will be called for every pair of points to check.
clustering.setConditionFunction(&customCondition);
std::vector<pcl::PointIndices> clusters;
clustering.segment(clusters);

// For every cluster...
int currentClusterNum = 1;
for (std::vector<pcl::PointIndices>::const_iterator i = clusters.begin(); i != clusters.end(); ++i)
{
// ...add all its points to a new cloud...
pcl::PointCloud<pcl::PointXYZ>::Ptr cluster(new pcl::PointCloud<pcl::PointXYZ>);
for (std::vector<int>::const_iterator point = i->indices.begin(); point != i->indices.end(); point++)
cluster->points.push_back(cloud->points[*point]);
cluster->width = cluster->points.size();
cluster->height = 1;
cluster->is_dense = true;

// ...and save it to disk.
if (cluster->points.size() <= 0)
break;
std::cout << "Cluster " << currentClusterNum << " has " << cluster->points.size() << " points." << std::endl;
std::string fileName = "cluster" + boost::to_string(currentClusterNum) + ".pcd";
pcl::io::savePCDFileASCII(fileName, *cluster);

currentClusterNum++;
}
}</syntaxhighlight>

The condition that is implemented above (checking if the candidate's Y coordinate is lower than the seed's) does not make a lot of sense, but I just wanted you to understand how the method works.

If you pass <span style="color:#FF1493">"true"</span> to the constructor of <span style="color:#FF1493">"pcl::ConditionalEuclideanClustering"</span>, the object will store the clusters that were discarded for being too small or too big. You can later retrieve them with the [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_euclidean_clustering.html#af264099e843fbf5d48049857e1e1d6af getRemovedClusters()] function.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Cluster tolerance, Minimum cluster size, Maximum cluster size, Search method, Custom check function, [Indices]
* '''Output''': Vector of Point indices (clusters)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/conditional_euclidean_clustering.php Conditional Euclidean Clustering]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_euclidean_clustering.html pcl::ConditionalEuclideanClustering]
* [http://robotica.unileon.es/~victorm/PCL_conditional_euclidean_clustering.tar.gz Download]
</div>

==Region growing==

This type of segmentation (which is also a greedy-like, flood fill approach like the Euclidean one) groups together points that check a smoothness constraint. The angle between their normals and the difference of curvatures are checked to see if they could belong to the same smooth surface. Think about a box laying on a table: with Euclidean segmentation, both would be considered to be in the same cluster because they are "touching". With region growing segmentation, this would not happen, because there is a 90° (with ideal normal estimation, that is) difference between the normals of a point in the table and another one in the box's lateral.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>
#include <pcl/features/normal_3d.h>
#include <pcl/segmentation/region_growing.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object for searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
kdtree->setInputCloud(cloud);

// Estimate the normals.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// Region growing clustering object.
pcl::RegionGrowing<pcl::PointXYZ, pcl::Normal> clustering;
clustering.setMinClusterSize(100);
clustering.setMaxClusterSize(10000);
clustering.setSearchMethod(kdtree);
clustering.setNumberOfNeighbours(30);
clustering.setInputCloud(cloud);
clustering.setInputNormals(normals);
// Set the angle in radians that will be the smoothness threshold
// (the maximum allowable deviation of the normals).
clustering.setSmoothnessThreshold(7.0 / 180.0 * M_PI); // 7 degrees.
// Set the curvature threshold. The disparity between curvatures will be
// tested after the normal deviation check has passed.
clustering.setCurvatureThreshold(1.0);

std::vector <pcl::PointIndices> clusters;
clustering.extract(clusters);

// For every cluster...
int currentClusterNum = 1;
for (std::vector<pcl::PointIndices>::const_iterator i = clusters.begin(); i != clusters.end(); ++i)
{
// ...add all its points to a new cloud...
pcl::PointCloud<pcl::PointXYZ>::Ptr cluster(new pcl::PointCloud<pcl::PointXYZ>);
for (std::vector<int>::const_iterator point = i->indices.begin(); point != i->indices.end(); point++)
cluster->points.push_back(cloud->points[*point]);
cluster->width = cluster->points.size();
cluster->height = 1;
cluster->is_dense = true;

// ...and save it to disk.
if (cluster->points.size() <= 0)
break;
std::cout << "Cluster " << currentClusterNum << " has " << cluster->points.size() << " points." << std::endl;
std::string fileName = "cluster" + boost::to_string(currentClusterNum) + ".pcd";
pcl::io::savePCDFileASCII(fileName, *cluster);

currentClusterNum++;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Minimum cluster size, Maximum cluster size, Neighbor count, Search method, Normals, Smoothness threshold, Curvature threshold, [Indices]
* '''Output''': Vector of Point indices (clusters)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/region_growing_segmentation.php Region growing segmentation]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_region_growing.html pcl::RegionGrowing]
* [http://robotica.unileon.es/~victorm/PCL_region_growing_clustering.tar.gz Download]
</div>

===Color-based===

This method is based on the previous one but, instead of comparing the normals and the curvature of the points, it compares the RGB color. Like always, if the difference is less than a threshold, both are put in the same cluster. Obviously, for this to work you need a cloud with RGB information (with the type <span style="color:#FF1493">"PointXYZRGB"</span>). You can use the [[PCL/OpenNI_tutorial_1:_Installing_and_testing#Testing_(OpenNI_viewer) | OpenNI viewer]] program to retrieve them with the Kinect or Xtion.

Apart from the color check, this algorithm also uses two postprocessing steps. In the first one, neighboring clusters with similar average color are merged (this is controlled with a threshold, too). In the second one, clusters that are smaller than the minimum size are merged with their neighbors.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>
#include <pcl/segmentation/region_growing_rgb.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud, with color information.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object for searches.
pcl::search::KdTree<pcl::PointXYZRGB>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZRGB>);
kdtree->setInputCloud(cloud);

// Color-based region growing clustering object.
pcl::RegionGrowingRGB<pcl::PointXYZRGB> clustering;
clustering.setInputCloud(cloud);
clustering.setSearchMethod(kdtree);
// Here, the minimum cluster size affects also the postprocessing step:
// clusters smaller than this will be merged with their neighbors.
clustering.setMinClusterSize(100);
// Set the distance threshold, to know which points will be considered neighbors.
clustering.setDistanceThreshold(10);
// Color threshold for comparing the RGB color of two points.
clustering.setPointColorThreshold(6);
// Region color threshold for the postprocessing step: clusters with colors
// within the threshold will be merged in one.
clustering.setRegionColorThreshold(5);

std::vector <pcl::PointIndices> clusters;
clustering.extract(clusters);

// For every cluster...
int currentClusterNum = 1;
for (std::vector<pcl::PointIndices>::const_iterator i = clusters.begin(); i != clusters.end(); ++i)
{
// ...add all its points to a new cloud...
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cluster(new pcl::PointCloud<pcl::PointXYZRGB>);
for (std::vector<int>::const_iterator point = i->indices.begin(); point != i->indices.end(); point++)
cluster->points.push_back(cloud->points[*point]);
cluster->width = cluster->points.size();
cluster->height = 1;
cluster->is_dense = true;

// ...and save it to disk.
if (cluster->points.size() <= 0)
break;
std::cout << "Cluster " << currentClusterNum << " has " << cluster->points.size() << " points." << std::endl;
std::string fileName = "cluster" + boost::to_string(currentClusterNum) + ".pcd";
pcl::io::savePCDFileASCII(fileName, *cluster);

currentClusterNum++;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Minimum cluster size, Maximum cluster size, Search method, Distance threshold, Point color threshold, Region color threshold, [Indices]
* '''Output''': Vector of Point indices (clusters)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/region_growing_rgb_segmentation.php Color-based region growing segmentation]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_region_growing_r_g_b.html pcl::RegionGrowingRGB]
* [http://robotica.unileon.es/~victorm/PCL_color_region_growing_clustering.tar.gz Download]
</div>

==Min-Cut==

The Min-Cut (minimum cut) algorithm is different from the previous ones. It performs binary segmentation: it divides the cloud in two clusters, one with points that do not belong to the object we are interested in (background points) and another with points that are considered part of the object (foreground points).

In order to do that, the algorithm uses a vertex graph, where each vertex (node of the graph) represents a point, plus two additional vertices called ''source'' and ''sink'' that will be connected to the others with edges with different penalties (weights). Then, edges are also established between neighboring points, with a weight value each that depends on the distance that separates them. The greater the distance, the higher the probability of the edge being cut. The algorithm also needs as input a point that is known to be the center of the object, and the radius (size). After everything has been set, the minimum cut will be found, and we will have a list of foreground points.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/segmentation/min_cut_segmentation.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Min-cut clustering object.
pcl::MinCutSegmentation<pcl::PointXYZ> clustering;
clustering.setInputCloud(cloud);
// Create a cloud that lists all the points that we know belong to the object
// (foreground points). We should set here the object's center.
pcl::PointCloud<pcl::PointXYZ>::Ptr foregroundPoints(new pcl::PointCloud<pcl::PointXYZ>());
pcl::PointXYZ point;
point.x = 100.0;
point.y = 100.0;
point.z = 100.0;
foregroundPoints->points.push_back(point);
clustering.setForegroundPoints(foregroundPoints);
// Set sigma, which affects the smooth cost calculation. It should be
// set depending on the spacing between points in the cloud (resolution).
clustering.setSigma(0.05);
// Set the radius of the object we are looking for.
clustering.setRadius(0.20);
// Set the number of neighbors to look for. Increasing this also increases
// the number of edges the graph will have.
clustering.setNumberOfNeighbours(20);
// Set the foreground penalty. It is the weight of the edges
// that connect clouds points with the source vertex.
clustering.setSourceWeight(0.6);

std::vector <pcl::PointIndices> clusters;
clustering.extract(clusters);

std::cout << "Maximum flow is " << clustering.getMaxFlow() << "." << std::endl;

// For every cluster...
int currentClusterNum = 1;
for (std::vector<pcl::PointIndices>::const_iterator i = clusters.begin(); i != clusters.end(); ++i)
{
// ...add all its points to a new cloud...
pcl::PointCloud<pcl::PointXYZ>::Ptr cluster(new pcl::PointCloud<pcl::PointXYZ>);
for (std::vector<int>::const_iterator point = i->indices.begin(); point != i->indices.end(); point++)
cluster->points.push_back(cloud->points[*point]);
cluster->width = cluster->points.size();
cluster->height = 1;
cluster->is_dense = true;

// ...and save it to disk.
if (cluster->points.size() <= 0)
break;
std::cout << "Cluster " << currentClusterNum << " has " << cluster->points.size() << " points." << std::endl;
std::string fileName = "cluster" + boost::to_string(currentClusterNum) + ".pcd";
pcl::io::savePCDFileASCII(fileName, *cluster);

currentClusterNum++;
}
}</syntaxhighlight>

More details about the min-cut algorithm can be found in the [http://gfx.cs.princeton.edu/pubs/Golovinskiy_2009_MBS/paper_small.pdf original paper (PDF)].

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Foreground points (object's center), Object's radius, Sigma, Neighbor count, Foreground penalty, [Indices]
* '''Output''': Vector of Point indices (object cluster)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/min_cut_segmentation.php Min-Cut Based Segmentation]
* '''Publication''':
** [http://gfx.cs.princeton.edu/pubs/Golovinskiy_2009_MBS/paper_small.pdf Min-Cut Based Segmentation of Point Clouds] (Aleksey Golovinskiy and Thomas Funkhouser, 2009)
* [http://robotica.unileon.es/~victorm/PCL_min_cut_clustering.tar.gz Download]
</div>

==RANSAC==

As we saw in a previous section, PCL offers an implementation of the RANSAC algorithm to fit a set of points to a model. It will search the cloud and find a list of points that support the chosen model (plane, sphere...), and also compute the coefficients of the model. An object exists to perform easy segmentation of the points retrieved by the algorithm.

If what you want is to extract the points of a model with already known coefficients, see this tutorial:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/model_outlier_removal.php Filtering a PointCloud using ModelOutlierRemoval]
</div>

===Plane model===

The following code lets you segment all planar surfaces from a point cloud. This is very important, because you will be able to detect things like the floor, ceiling, walls, or a table in a scene. If you know that the object you are looking for is sitting on a table, you can discard all points that are not supported by it, with the associated performance gain. Take a look at the [[PCL/OpenNI_tutorial_5:_3D_object_recognition_%28pipeline%29#Segmentation|pcl::ExtractPolygonalPrismData]] class, which is designed for this. After giving the coefficients of the plane and computing the convex hull, you can then create a 3D prism by extruding the hull a given height, and extract all points that lie inside it.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr inlierPoints(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for storing the plane model coefficients.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
// Create the segmentation object.
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
// Configure the object to look for a plane.
segmentation.setModelType(pcl::SACMODEL_PLANE);
// Use RANSAC method.
segmentation.setMethodType(pcl::SAC_RANSAC);
// Set the maximum allowed distance to the model.
segmentation.setDistanceThreshold(0.01);
// Enable model coefficient refinement (optional).
segmentation.setOptimizeCoefficients(true);

pcl::PointIndices inlierIndices;
segmentation.segment(inlierIndices, *coefficients);

if (inlierIndices.indices.size() == 0)
std::cout << "Could not find any points that fitted the plane model." << std::endl;
else
{
std::cerr << "Model coefficients: " << coefficients->values[0] << " "
<< coefficients->values[1] << " "
<< coefficients->values[2] << " "
<< coefficients->values[3] << std::endl;

// Copy all inliers of the model to another cloud.
pcl::copyPointCloud<pcl::PointXYZ>(*cloud, inlierIndices, *inlierPoints);
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Model type, Method type, Model threshold, [Optimize coefficients]
* '''Output''': Vector of Point indices (plane cluster)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/planar_segmentation.php Plane model segmentation]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_s_a_c_segmentation.html pcl::SACSegmentation]
* [http://robotica.unileon.es/~victorm/PCL_plane_segmentation.tar.gz Download]
</div>

===Cylinder model===

Segmenting a cylinder shape out of the cloud if analogous. There is only an additional parameter for specifying the radii:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr inlierPoints(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for storing the plane model coefficients.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
// Create the segmentation object.
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
// Configure the object to look for a plane.
segmentation.setModelType(pcl::SACMODEL_CYLINDER);
// Use RANSAC method.
segmentation.setMethodType(pcl::SAC_RANSAC);
// Set the maximum allowed distance to the model.
segmentation.setDistanceThreshold(0.01);
// Enable model coefficient refinement (optional).
segmentation.setOptimizeCoefficients(true);
// Set minimum and maximum radii of the cylinder.
segmentation.setRadiusLimits(0, 0.1);

pcl::PointIndices inlierIndices;
segmentation.segment(inlierIndices, *coefficients);

if (inlierIndices.indices.size() == 0)
std::cout << "Could not find any points that fitted the cylinder model." << std::endl;
// Copy all inliers of the model to another cloud.
else pcl::copyPointCloud<pcl::PointXYZ>(*cloud, inlierIndices, *inlierPoints);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Model type, Method type, Model threshold, [Optimize coefficients]
* '''Output''': Vector of Point indices (cylinder cluster)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/cylinder_segmentation.php Cylinder model segmentation]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_s_a_c_segmentation.html pcl::SACSegmentation]
* [http://robotica.unileon.es/~victorm/PCL_cylinder_segmentation.tar.gz Download]
</div>

=Retrieving the hull=

For a given set of points, PCL can compute the external hull of the shape, using the [http://www.qhull.org/ QHull library]. The hull could be defined as the points that conform the outermost boundary of the set, like a shell showing the volume. There are two types of hull that you can calculate, the [https://en.wikipedia.org/wiki/Convex_hull convex] and the concave one. A convex hull can not have "holes", that is, the segment that connects any pair of points can never cross "empty" space (not inside the hull). A concave hull, on the other hand, usually takes less area, like you can see in the following images:

<center><gallery widths=300px>
File:ConcaveHull.png | Concave hull.
File:ConvexHull.png | Convex hull.
</gallery></center>

==Concave hull==

The following code shows how you can use PCL to compute the concave hull of the first plane found in the scene:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/surface/concave_hull.h>
#include <pcl/visualization/cloud_viewer.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr plane(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr concaveHull(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Get the plane model, if present.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setOptimizeCoefficients(true);
pcl::PointIndices::Ptr inlierIndices(new pcl::PointIndices);
segmentation.segment(*inlierIndices, *coefficients);

if (inlierIndices->indices.size() == 0)
std::cout << "Could not find a plane in the scene." << std::endl;
else
{
// Copy the points of the plane to a new cloud.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloud);
extract.setIndices(inlierIndices);
extract.filter(*plane);

// Object for retrieving the concave hull.
pcl::ConcaveHull<pcl::PointXYZ> hull;
hull.setInputCloud(plane);
// Set alpha, which is the maximum length from a vertex to the center of the voronoi cell
// (the smaller, the greater the resolution of the hull).
hull.setAlpha(0.1);
hull.reconstruct(*concaveHull);

// Visualize the hull.
pcl::visualization::CloudViewer viewerPlane("Concave hull");
viewerPlane.showCloud(concaveHull);
while (!viewerPlane.wasStopped())
{
// Do nothing but wait.
}
}
}</syntaxhighlight>

By choosing a smaller alpha value, you can improve the detail of the hull.

<center><gallery widths=300px>
File:ConcaveHullBefore.png | Table scene taken from the PCL dataset.
File:ConcaveHullAfter.png | Concave hull of the table seen in the cloud (better seen fullscreen).
</gallery></center>

The class [http://docs.pointclouds.org/trunk/classpcl_1_1_crop_hull.html pcl::CropHull] lets you find points that lie inside or outside a hull. One possible use, for a scene with a table and objects on it, would be:

# Find a plane in the scene (it should be the table).
# Compute the hull of the plane.
# Project the points of the cloud onto the plane.
# Find which projected points are lying inside the plane hull. Those belong to objects sitting on the table.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Alpha
* '''Output''': Hull points (concave)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/hull_2d.php Construct a concave or convex hull polygon for a plane model]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_concave_hull.html pcl::ConcaveHull]
* [http://robotica.unileon.es/~victorm/PCL_concave_hull.tar.gz Download]
</div>

==Convex hull==

Computing the convex hull is done the same way, just changing the names accordingly. Also, there is no need to set the alpha parameter here:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/surface/convex_hull.h>
#include <pcl/visualization/cloud_viewer.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr plane(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr convexHull(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Get the plane model, if present.
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setInputCloud(cloud);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setOptimizeCoefficients(true);
pcl::PointIndices::Ptr inlierIndices(new pcl::PointIndices);
segmentation.segment(*inlierIndices, *coefficients);

if (inlierIndices->indices.size() == 0)
std::cout << "Could not find a plane in the scene." << std::endl;
else
{
// Copy the points of the plane to a new cloud.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloud);
extract.setIndices(inlierIndices);
extract.filter(*plane);

// Object for retrieving the convex hull.
pcl::ConvexHull<pcl::PointXYZ> hull;
hull.setInputCloud(plane);
hull.reconstruct(*convexHull);

// Visualize the hull.
pcl::visualization::CloudViewer viewerPlane("Convex hull");
viewerPlane.showCloud(convexHull);
while (!viewerPlane.wasStopped())
{
// Do nothing but wait.
}
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points
* '''Output''': Hull points (convex)
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/hull_2d.php Construct a concave or convex hull polygon for a plane model]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_convex_hull.html pcl::ConvexHull]
* [http://robotica.unileon.es/~victorm/PCL_convex_hull.tar.gz Download]
</div>

=Reconstruction=

To learn about how to use PCL to build a smooth, parametric surface representation from a point cloud, you can read the following:

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/bspline_fitting.php Fitting trimmed B-splines to unordered point clouds]
</div>

==Triangulation==

Triangulation is a a way of estimating the surface captured by a point cloud, by connecting points with each other, ending up with a continous polygon mesh (three-sided polygons, that is, triangles). After you retrieve this surface mesh, you can for example export it to a format that most 3D modelling tools will understand, like [https://en.wikipedia.org/wiki/VTK VTK (Visualization Toolkit)], which can be opened in Blender or 3ds Max. PCL can also convert from VTK to [https://en.wikipedia.org/wiki/PLY_%28file_format%29 PLY] or [https://en.wikipedia.org/wiki/Wavefront_.obj_file OBJ].

The following code uses PCL's implementation of a greedy triangulation algorithm that works with local 2D projections. For every point, it looks "down" along the normal, and connects neighboring points. For more information, specially regarding the parameters, check the API or the original PCL tutorial:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <pcl/surface/gp3.h>
#include <pcl/io/vtk_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
// Object for storing both the points and the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr cloudNormals(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
normalEstimation.setRadiusSearch(0.03);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*normals);

// The triangulation object requires the points and normals to be stored in the same structure.
pcl::concatenateFields(*cloud, *normals, *cloudNormals);
// Tree object for searches in this new object.
pcl::search::KdTree<pcl::PointNormal>::Ptr kdtree2(new pcl::search::KdTree<pcl::PointNormal>);
kdtree2->setInputCloud(cloudNormals);

// Triangulation object.
pcl::GreedyProjectionTriangulation<pcl::PointNormal> triangulation;
// Output object, containing the mesh.
pcl::PolygonMesh triangles;
// Maximum distance between connected points (maximum edge length).
triangulation.setSearchRadius(0.025);
// Maximum acceptable distance for a point to be considered,
// relative to the distance of the nearest point.
triangulation.setMu(2.5);
// How many neighbors are searched for.
triangulation.setMaximumNearestNeighbors(100);
// Points will not be connected to the current point
// if their normals deviate more than the specified angle.
triangulation.setMaximumSurfaceAngle(M_PI / 4); // 45 degrees.
// If false, the direction of normals will not be taken into account
// when computing the angle between them.
triangulation.setNormalConsistency(false);
// Minimum and maximum angle there can be in a triangle.
// The first is not guaranteed, the second is.
triangulation.setMinimumAngle(M_PI / 18); // 10 degrees.
triangulation.setMaximumAngle(2 * M_PI / 3); // 120 degrees.

// Triangulate the cloud.
triangulation.setInputCloud(cloudNormals);
triangulation.setSearchMethod(kdtree2);
triangulation.reconstruct(triangles);

// Save to disk.
pcl::io::saveVTKFile("mesh.vtk", triangles);
}</syntaxhighlight>

[[Image:VTK.png|thumb|center|600px|VTK file produced by triangulation of a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Maximum distance, Maximum edge length, Minimum angle, Maximum angle, Maximum surface angle, [Normal consistency]
* '''Output''': Polygon mesh
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/greedy_projection.php Fast triangulation of unordered point clouds]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1_greedy_projection_triangulation.html pcl::GreedyProjectionTriangulation]
** [http://docs.pointclouds.org/trunk/structpcl_1_1_polygon_mesh.html pcl::PolygonMesh]
* [http://robotica.unileon.es/~victorm/PCL_triangulation.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-01T23:05:50Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [https://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, "reference", 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [https://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [https://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [https://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [https://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [https://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [https://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://en.wikipedia.org/wiki/CUDA CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [https://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-01T23:04:43Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [https://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, "reference", 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [https://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [https://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [https://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [https://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [https://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [https://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://developer.nvidia.com/about-cuda CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [https://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [https://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-01T23:02:09Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from the [https://en.wikipedia.org/wiki/Boost_%28C%2B%2B_libraries%29 Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [http://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, "reference", 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [http://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [http://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [http://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [http://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [http://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [http://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://developer.nvidia.com/about-cuda CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [http://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-01T23:01:21Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [https://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from [http://www.boost.org/ Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [http://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, "reference", 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [http://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [http://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [http://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [http://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [http://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [http://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://developer.nvidia.com/about-cuda CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [http://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 2: Cloud processing (basic)

2015-11-01T23:00:38Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Point_cloud.png|thumb|right|200px|Rendering of a point cloud, showing a desktop.]]

All right: in the [[PCL/OpenNI tutorial 1: Installing and testing| previous tutorial]] you installed OpenNI and PCL. You managed to get your device working. You compiled and ran the example program, watched as it fetched one frame after another, and even saved a couple of point clouds to disk. Now you may be wondering "What about now? What use depth sensors are? What can I do with point clouds?"

Well, depth cameras have a lot of useful applications. For example, you could use them as [http://tech.integrate.biz/kinect_mocap.htm motion tracking] hardware and save you thousands of dollars in professional gear. The original Kinect was actually designed as an alternative to physical controllers (e.g., gamepads) in games. [https://frantracerkinectft.codeplex.com/ Hand gestures] can also be recognized, so people have managed to successfully implement control interfaces similar to what could be seen in Minority Report.

Working low-level with point clouds offers interesting possibilities too. As they are essentially a 3D "scan" or the scene, you could capture your whole room as a [http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php continuous 3D mesh], then import it in your modelling software of choice and use it as a reference for a better reconstruction. You could use it to measure distances, or detect obstacles, for a robot navigation system. But the most exciting alternative by far, and the one where most of the research is being focused on, is to perform 3D object recognition and pose estimation, even in [http://www.youtube.com/watch?feature=player_detailpage&v=quGhaggn3cQ#t=349 real time]. This allows a robot to fulfill tasks like going to the kitchen to get you a cup of coffee.

Most of the advanced stuff that can be done with a point cloud requires some previous steps, like filtering, reconstruction or normal estimation. In this tutorial, I will introduce you to the basics of point cloud processing, and leave the complicated methods for the [[PCL/OpenNI tutorial 3: Cloud processing (advanced) | next tutorial]]. I will explain what every technique does and what it should be used for, and include PCL code snippets so you can check how to implement it in your programs.

=Point clouds=

First, a little explanation. A point cloud as taken from a depth sensor consists of a series of points in 3D space, as simple as that. A <span style="color:#FF1493">"pcl::PointCloud<PointT>"</span> object stores the points inside a <span style="color:#FF1493">"std::vector<PointT>"</span> structure. The cloud object exposes some functions that let you get information about them, like the point count. As you can see, the class is templated so you can instantiate it for many types of points. For example, <span style="color:#FF1493">"PointXYZ"</span>, which stores three floats for the X, Y and Z coordinates, and <span style="color:#FF1493">"PointXYZRGB"</span>, which also stores color (texture) information for each point, the kind we would get with Kinect or Xtion RGB-D cameras. A list of [http://pointclouds.org/documentation/tutorials/adding_custom_ptype.php#what-pointt-types-are-available-in-pcl point types] and a basic description of the [http://pointclouds.org/documentation/tutorials/basic_structures.php cloud object] are available on PCL's website.

Clouds will usually be handled with pointers, as they tend to be large. PCL implements the <span style="color:#FF1493">"boost::shared_ptr"</span> class from [http://www.boost.org/ Boost C++ libraries], that keeps track of the references to a certain object. Every time you pass a shared pointer and a copy of it is created, a reference counter is automatically increased. When the pointer is destroyed, it is decreased. If the counter ever reaches 0, it means that no one is holding an active reference to the object, and its destructor is called to free memory. This can take a while to get used to, and you will be mixing cloud objects and cloud object pointers until then, but do not worry.

==PCD files==

When clouds are saved to disk, a [http://pointclouds.org/documentation/tutorials/pcd_file_format.php PCD] (Point Cloud Data) file is produced. You can choose to do it so in binary or plain text format. The first one is faster to work with, whereas the second one will let you to inspect the cloud by opening it with a common text editor. If you do so, you may see something like this:

<syntaxhighlight lang=CPP># .PCD v.7 − Point Cloud Data file format
VERSION .7
FIELDS x y z
SIZE 4 4 4
TYPE F F F
WIDTH 2
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 2
DATA ascii
0.73412 -1.12643 0.82218
0.44739 -0.34735 -0.04624</syntaxhighlight>

This tells us that the cloud consist of 2 points of type <span style="color:#FF1493">"PointXYZ"</span> (because each point has only 3 fields; the X, Y and Z coordinates, each stored in a 4-byte float), and lists their coordinates. It also stores the viewport information (the relative position of the points to the sensor) as a translation quaternion. Knowing the position of the camera can be useful for certain procedures like normal estimation.

===Reading from file===

Reading a point cloud from a PCD file to and storing it in memory can be done with a single function call. You must specify the correct point type. For example, for .pcd files saved with the OpenNI viewer application that we saw in the [[PCL/OpenNI tutorial 1: Installing and testing | previous tutorial]], you could use either <span style="color:#FF1493">"PointXYZRGBA"</span> or <span style="color:#FF1493">"PointXYZ"</span> if you are not going to use the color information.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}
}</syntaxhighlight>

I have not bothered to include proper error/argument checking, but you should do it in your programs. Be sure to check the [http://docs.pointclouds.org PCL API] every time I introduce you to one of its classes or methods.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/reading_pcd.php Reading Point Cloud data from PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_read.tar.gz Download]
</div>

===Writing to file===

Saving a point cloud to a file is as easy as reading it. You can choose between doing it in plain readable (ASCII) or binary format. The first one can be read and altered with a text editor, the second one is faster.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Write it back to disk under a different name.
// Another possibility would be "savePCDFileBinary()".
pcl::io::savePCDFileASCII("output.pcd", *cloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/writing_pcd.php Writing Point Cloud data to PCD files]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl_1_1io.html pcl::io namespace]
* [http://robotica.unileon.es/~victorm/PCL_file_write.tar.gz Download]
</div>

If you want a better alternative for loading or saving PCD files (some of the functions we saw are a bit outdated, but they do the job), take a look at the [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_reader.html PCDReader] and [http://docs.pointclouds.org/trunk/classpcl_1_1_p_c_d_writer.html PCDWriter] classes.

==Visualization==

Of course, PCL ships with a dedicated program to load and visualize point clouds and a lot of other stuff, <span style="color:#228B22">''pcl_viewer''</span>. You can run it from a terminal with:

<syntaxhighlight lang=Bash>pcl_viewer <file 1> [file 2] ... [file N]</syntaxhighlight>

If you load several files, each cloud will be rendered with a different color to help you tell them apart. The program uses internally the [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_p_c_l_visualizer.html pcl::visualization::PCLVisualizer] class. Press the <span style="color:#228B22">'''H'''</span> key while it is running to print the help on the terminal. Keys I tend to find useful are <span style="color:#228B22">'''1'''</span>, which chooses a different color for the cloud (useful if the current one is too hard to see), <span style="color:#228B22">'''R'''</span>, which resets the camera, and <span style="color:#228B22">'''J'''</span>, which saves a PNG snapshot.

In your own code, it is possible to iterate through the cloud and print the information of each point to console, but this is cumbersome. Sometimes you just want to visualize the cloud to check something. PCL can create a window for you and display it, letting users pan and zoom their way around. The simplest method is the following one:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Cloud viewer object. You can set the window title.
pcl::visualization::CloudViewer viewer(argv[1]);
viewer.showCloud(cloud);
while (!viewer.wasStopped())
{
// Do nothing but wait.
}
}</syntaxhighlight>

<span style="color:#FF1493">"CloudViewer"</span> has less features than <span style="color:#FF1493">"PCLVisualizer"</span> but it is easier to set up. The program will remain active until you close the window with <span style="color:#228B22">'''Alt+F4'''</span> or press <span style="color:#228B22">'''Q'''</span>.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorials''':
** [http://pointclouds.org/documentation/tutorials/cloud_viewer.php The CloudViewer]
** [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer]
** [http://pointclouds.org/documentation/tutorials/walkthrough.php Visualization library overview]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1visualization_1_1_cloud_viewer.html pcl::visualization::CloudViewer]
* [http://robotica.unileon.es/~victorm/PCL_visualize_cloud.tar.gz Download]
</div>

==Concatenating two clouds==

There are two types of concatenation with point clouds:

* '''Points''': You initially have two clouds, "A" and "B", with "N" and "M" points each. A third cloud is created ("C"), that has "N + M" points (all points of the first two clouds). For this to work, the point type must be the same for all clouds (<span style="color:#FF1493">"PointXYZ"</span>, or <span style="color:#FF1493">"Normal"</span>, or whatever).
* '''Fields''': With this, you can join the fields of two clouds with different types. The only constraint is that both must have the same number of points. For example, if you have a cloud of the <span style="color:#FF1493">"PointXYZ"</span> type and another of <span style="color:#FF1493">"Normal"</span> type with "N" points each, you can concatenate them to create a cloud of type <span style="color:#FF1493">"PointNormal"</span> (that stores X, Y and Z coordinates, plus the normal) that also has "N" points.

Point concatenation is the easiest one becase you can use the cloud's add operator (+), which is overloaded.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloudC(new pcl::PointCloud<pcl::PointXYZRGBA>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZRGBA>(argv[2], *cloudB) != 0)
{
return -1;
}

// Create cloud "C", with the points of both "A" and "B".
*cloudC = (*cloudA) + (*cloudB);
// Now cloudC->points.size() equals cloudA->points.size() + cloudB->points.size().
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_clouds.tar.gz Download]
</div>

Field concatenation is done with a function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudPoints(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::Normal>::Ptr cloudNormals(new pcl::PointCloud<pcl::Normal>);
pcl::PointCloud<pcl::PointNormal>::Ptr cloudAll(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudPoints) != 0)
{
return -1;
}

// Compute the normals of the cloud (do not worry, we will see this later).
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloudPoints);
normalEstimation.setRadiusSearch(0.05);
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);
normalEstimation.compute(*cloudNormals);

// Concatenate the fields (PointXYZ + Normal = PointNormal).
pcl::concatenateFields(*cloudPoints, *cloudNormals, *cloudAll);

// Print the data to standard output.
for (size_t currentPoint = 0; currentPoint < cloudAll->points.size(); currentPoint++)
{
std::cout << "Point:" << std::endl;
std::cout << "\tXYZ:" << cloudAll->points[currentPoint].x << " "
<< cloudAll->points[currentPoint].y << " "
<< cloudAll->points[currentPoint].z << std::endl;
std::cout << "\tNormal:" << cloudAll->points[currentPoint].normal[0] << " "
<< cloudAll->points[currentPoint].normal[1] << " "
<< cloudAll->points[currentPoint].normal[2] << std::endl;
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/concatenate_clouds.php Concatenate the points of two Point Clouds]
* '''API''': [http://docs.pointclouds.org/trunk/group__common.html#gac6add803f86fd16a998dce541e9ef402 pcl::concatenateFields()]
* [http://robotica.unileon.es/~victorm/PCL_concatenate_fields.tar.gz Download]
</div>

==Matrix transformations==

Clouds can be directly manipulated with [http://en.wikipedia.org/wiki/Transformation_matrix transformations] (translation, rotation, scaling). 3D transformations use a 4x4 matrix, which can be a little complicated to understand (specially when it comes to rotations), so check some book or [http://web.iitd.ac.in/~hegde/cad/lecture/L6_3dtrans.pdf tutorial] before trying to use one. The transformed cloud will be saved to another object so you can keep both versions and use them as you wish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr transformed(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Transformation matrix object, initialized to the identity matrix
// (a null transformation).
Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();

// Set a rotation around the Z axis.
float theta = M_PI / 2; // 180 degrees.
transformation(0, 0) = cos(theta);
transformation(0, 1) = -sin(theta);
transformation(1, 0) = sin(theta);
transformation(1, 1) = cos(theta);

// Set a translation on the X axis.
transformation(0, 3) = 1.0f; // 1 meter (positive direction).

pcl::transformPointCloud(*cloud, *transformed, transformation);

// Visualize both the original and the result.
pcl::visualization::PCLVisualizer viewer(argv[1]);
viewer.addPointCloud(cloud, "original");
// The transformed one's points will be red in color.
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> colorHandler(transformed, 255, 0, 0);
viewer.addPointCloud(transformed, colorHandler, "transformed");
// Add 3D colored axes to help see the transformation.
viewer.addCoordinateSystem(1.0, "reference", 0);

while (!viewer.wasStopped())
{
viewer.spinOnce();
}
}</syntaxhighlight>

[[Image:Cloud_transformation.png|thumb|center|600px|Rotation and translation applied to a cloud.]]

Check the rest of the functions, because you can also use a <span style="color:#FF1493">"Eigen::Affine3"</span> or a <span style="color:#FF1493">"Eigen::Quaternion"</span> instead of a <span style="color:#FF1493">"Eigen::Matrix4"</span> object.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Transformation
* '''Output''': Transformed points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/matrix_transform.php Using a matrix to transform a point cloud]
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#aefbe4956d1c8fb785a97df6708d57c56 pcl::transformPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_matrix_transform.tar.gz Download]
</div>

==Using indices==

Many of PCL's algorithms return indices. Indices are just a list of some of the points of a cloud (not the points themselves with all their data, but just their index in the cloud). For example, a cylinder segmentation algorithm will give you as output a list of points that were considered to fit the cylinder model. You can then use the indices to extract the points to a second cloud for further work with the <span style="color:#FF1493">"pcl::ExtractIndices"</span> class. Its <span style="color:#FF1493">"setNegative()"</span> function will instruct it to extract the points NOT indexed (so you can remove all cylinder-shaped clusters from the cloud).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/segmentation/sac_segmentation.h>

#include <vector>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudAll(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudExtracted(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudAll) != 0)
{
return -1;
}

// Plane segmentation (do not worry, we will see this later).
pcl::ModelCoefficients::Ptr coefficients(new pcl::ModelCoefficients);
pcl::SACSegmentation<pcl::PointXYZ> segmentation;
segmentation.setOptimizeCoefficients(true);
segmentation.setModelType(pcl::SACMODEL_PLANE);
segmentation.setMethodType(pcl::SAC_RANSAC);
segmentation.setDistanceThreshold(0.01);
segmentation.setInputCloud(cloudAll);

// Object for storing the indices.
pcl::PointIndices::Ptr pointIndices(new pcl::PointIndices);

segmentation.segment(*pointIndices, *coefficients);

// Object for extracting points from a list of indices.
pcl::ExtractIndices<pcl::PointXYZ> extract;
extract.setInputCloud(cloudAll);
extract.setIndices(pointIndices);
// We will extract the points that are NOT indexed (the ones that are not in a plane).
extract.setNegative(true);
extract.filter(*cloudExtracted);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Indices, [Negative mode]
* '''Output''': Points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/extract_indices.php Extracting indices from a PointCloud]
* '''API''':
** [http://docs.pointclouds.org/trunk/structpcl_1_1_point_indices.html pcl::PointIndices]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_extract_indices.html pcl::ExtractIndices]
* [http://robotica.unileon.es/~victorm/PCL_extract_indices.tar.gz Download]
</div>

==Removing NaNs==

The cloud retrieved from the sensor may contain several kind of measurement errors and/or inaccuracies. One of them is the presence of [http://en.wikipedia.org/wiki/NaN NaN (Not a Number)] values in the coordinates of some points, as you can see in the following file:

<syntaxhighlight lang=CPP># .PCD v0.7 - Point Cloud Data file format
VERSION 0.7
FIELDS x y z rgba
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 640
HEIGHT 480
VIEWPOINT 0 0 0 1 0 0 0
POINTS 307200
DATA ascii
nan nan nan 10135463
nan nan nan 10398635
nan nan nan 10070692
nan nan nan 10268071
...</syntaxhighlight>

Cloud objects have a member function called <span style="color:#FF1493">"is_dense()"</span> that will return true if all points have valid, finite values. A NaN indicates that there was a problem measuring the distance to that point, maybe because the sensor was too close or too far, or because the surface was reflective.

Sadly, many of PCL's algorithms will fail if even a single NaN is present in the cloud you feed them as input, with a message like <span style="color:#FF1493">''"Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!"' failed."''</span>. If this happens, you will have to remove such values from the cloud. You can use a program like the following to "clean" a cloud from all NaN points:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/filter.h>

#include <iostream>

int
main(int argc, char** argv)
{
if (argc != 3)
{
std::cout << "\tUsage: " << argv[0] << " <input cloud> <output cloud>" << std::endl;

return -1;
}

// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// The mapping tells you to what points of the old cloud the new ones correspond,
// but we will not use it.
std::vector<int> mapping;
pcl::removeNaNFromPointCloud(*cloud, *cloud, mapping);

// Save it back.
pcl::io::savePCDFileASCII(argv[2], *cloud);
}</syntaxhighlight>

The problem with this method is that it will not keep the cloud organized. All clouds store a <span style="color:#FF1493">"width"</span> and a <span style="color:#FF1493">"height"</span> variable. In ''unorganized'' clouds, the total number of points is the same as the width, and the height is set to 1. In ''organized'' clouds (like the ones taken from camera-like sensors such as Kinect or Xtion), the width and the height are the same that the pixels of the image resolution the sensor works with. Points are distributed in rows like in the depth image, and every one of them corresponds to a pixel. The member function <span style="color:#FF1493">"isOrganized()"</span> will return true if the height is greater than 1.

Because removing NaNs will change the number of points of the cloud, it can no longer be kept organized with the original width/height ratio, so the function will set the height to 1. This is not a big issue, as only a few of PCL's algorithms work explicitly with organized clouds (and they mostly use it for optimization), but you must take it into account.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Cloud
* '''Output''': Cloud without NaNs
* '''API''': [http://docs.pointclouds.org/trunk/group__filters.html#gac463283a9e9c18a66d3d29b28a575064 pcl::removeNaNFromPointCloud()]
* [http://robotica.unileon.es/~victorm/PCL_remove_NaNs.tar.gz Download]
</div>

==Saving a PNG==

PCL offers the possibility of saving the values of a point cloud to a PNG image file. Obviously, this can only be done with organized clouds, because the rows and columns of the resulting image will correspond exactly to the ones of the cloud. For example, if you have a point cloud taken from a sensor like Kinect or Xtion, you can use this to retrieve the 640x480 RGB image that matches the cloud.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/io/png_io.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZRGB>(argv[1], *cloud) != 0)
{
return -1;
}

// Save the image (cloud must be organized).
pcl::io::savePNGFile("output.png", *cloud, "rgb");
}</syntaxhighlight>

The function <span style="color:#FF1493">"savePNGFile()"</span> has several implementations; in this case, we have chosen the one that lets you specify which fields to save. If you omit that parameter, the function will save the RGB fields by default.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/group__io.html#gabd03e43fac3048635512f447c8bc50b3 pcl::io::savePNGFile()]
* [http://robotica.unileon.es/~victorm/PCL_save_PNG.tar.gz Download]
</div>

==Computing the centroid==

The [http://en.wikipedia.org/wiki/Centroid centroid] of a cloud is a point with coordinates that result from computing the mean of the values of all points in the cloud. You could say it is the "center of mass", and it has several uses for certain algorithms. If you intend to compute the actual center of gravity of a [[PCL/OpenNI_tutorial_3:_Cloud_processing_%28advanced%29#Segmentation|clustered]] object though, remember that the sensor does not retrieve the part of the object that is hidden from the camera, like the back faces that are occluded by the front ones, or the inside. You only have the part of the surface that faces the camera.

The centroid of a cloud can be computed with a single function call:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/common/centroid.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object to store the centroid coordinates.
Eigen::Vector4f centroid;

pcl::compute3DCentroid(*cloud, centroid);

std::cout << "The XYZ coordinates of the centroid are: ("
<< centroid[0] << ", "
<< centroid[1] << ", "
<< centroid[2] << ")." << std::endl;
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''API''': [http://docs.pointclouds.org/trunk/namespacepcl.html#a23daec3829d2d4100a2f185372b3753a pcl::compute3DCentroid()]
* [http://robotica.unileon.es/~victorm/PCL_compute_centroid.tar.gz Download]
</div>

=Feature estimation=

A ''feature'' of a point is some characteristic that can help us to tell it apart from a different point. Of course, you could say that the XYZ coordinates would be characteristic enough, but when interpolating the data of two different, yet corresponding, clouds, they become useless. We need something better, that will have similar values when computed for the same spot of a surface in both sets.

Normals are an example of a very simple feature. 3D [http://pointclouds.org/documentation/tutorials/how_features_work.php features] will be important later in our tutorials, when we talk about [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]] (more detailed "signatures" of points used for close matching).

==Normal estimation==

As you may remember from geometry class, the [http://en.wikipedia.org/wiki/Surface_normal normal] of a plane is an unit vector that is perpendicular to it. The normal of a surface at a point is defined as the vector that is perpendicular to the plane that is tangent to the surface at the point. Surface normals can be calculated for the points of a cloud, too. It is considered a feature, albeit not a very discriminative one.

I will not go into detail with the math of the estimation method, but you just have to know that is uses the nearest neighbors (the points that are closest to the one we are calculating the normal for) to find out the tangent plane and the normal vector. You can customize the method with the search radius (think about a sphere of that radius, centered in the point; all neighboring points that lie within will be used for the computation) and the viewpoint (by default, the output normals will be directionless; by supposing that all vectors must point towards the camera - because otherwise they would belong to surfaces that are not visible from the sensor - they can all be re-oriented accordingly).

Normals are also important because they give us information about the ''curvature'' of the surface at some point, which can be used to our advantage. Many of PCL's algorithms will require us to provide the normals of the input cloud. To estimate them, you can use the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/normal_3d.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::NormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// For every point, use all neighbors in a radius of 3cm.
normalEstimation.setRadiusSearch(0.03);
// A kd-tree is a data structure that makes searches efficient. More about it later.
// The normal estimation object will use it to find nearest neighbors.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree(new pcl::search::KdTree<pcl::PointXYZ>);
normalEstimation.setSearchMethod(kdtree);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

As you can see, normals are stored in <span style="color:#FF1493">"PointCloud"</span> objects too, instantiated to the <span style="color:#FF1493">"Normal"</span> type. If you ever extract some indices from a cloud before feeding them to an algorithm, be sure to do the same with the normals. I have not made use of the <span style="color:#FF1493">"setViewPoint()"</span> function, but you get the idea.

Instead of <span style="color:#FF1493">"setRadiusSearch()"</span>, you can make use of <span style="color:#FF1493">"setKSearch()"</span>, which takes an integer, ''K''. This will take into account a point's ''K'' nearest neighbors to compute the normal. Please mind that this can produce some odd results for points that are not in dense areas, as some of the "neighbors" will be too far from them.

[[Image:Normal_estimation.png|thumb|center|600px|Visualization of the normals computed for a point cloud.]]

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search method, Radius | Neighbor count, [Indices], [Viewpoint]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation.php Estimating Surface Normals in a Point Cloud]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_normal_estimation.html pcl::NormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals.tar.gz Download]
</div>

===Integral images===

Integral images are a method for normal estimation on organized clouds. The algorithm sees the cloud as a depth image, and creates certain rectangular areas over which the normals are computed, by taking into account the relationship between neighboring "pixels" (points), without the need to run lookups on a search structure like a tree. Because of this, it is very efficient and the normals are usually computed in a split second. If you are performing real time computations on clouds taken from a RGB-D sensor, this is the approach you should use because the previous method will probably take several seconds to finish.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/features/integral_image_normal.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// Object for storing the normals.
pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Object for normal estimation.
pcl::IntegralImageNormalEstimation<pcl::PointXYZ, pcl::Normal> normalEstimation;
normalEstimation.setInputCloud(cloud);
// Other estimation methods: COVARIANCE_MATRIX, AVERAGE_DEPTH_CHANGE, SIMPLE_3D_GRADIENT.
// They determine the smoothness of the result, and the running time.
normalEstimation.setNormalEstimationMethod(normalEstimation.AVERAGE_3D_GRADIENT);
// Depth threshold for computing object borders based on depth changes, in meters.
normalEstimation.setMaxDepthChangeFactor(0.02f);
// Factor that influences the size of the area used to smooth the normals.
normalEstimation.setNormalSmoothingSize(10.0f);

// Calculate the normals.
normalEstimation.compute(*normals);

// Visualize them.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Normals"));
viewer->addPointCloud<pcl::PointXYZ>(cloud, "cloud");
// Display one normal out of 20, as a line of length 3cm.
viewer->addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(cloud, normals, 20, 0.03, "normals");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Estimation method, [Smoothing factor], [Depth threshold], [Rectangle size], [Border policy], [Depth dependent smoothing]
* '''Output''': Normals
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/normal_estimation_using_integral_images.php Normal Estimation Using Integral Images]
* '''Publications''':
** [http://islab.ulsan.ac.kr/files/announcement/411/Adaptive%20Neighborhood%20Selection%20for%20Real-Time%20Surface%20Normal.pdf Adaptive Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images] (S. Holzer et al., 2012)
** [https://www.willowgarage.com/sites/default/files/holz_rgbd_seg.pdf Real-Time Plane Segmentation using RGB-D Cameras] (Dirk Holz et al., 2012)
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_integral_image_normal_estimation.html pcl::IntegralImageNormalEstimation]
* [http://robotica.unileon.es/~victorm/PCL_estimate_normals_integral.tar.gz Download]
</div>

=Decomposition=

When I talk about ''decomposition'', I mean the process of creating an ordered, organized data structure from the points of the cloud, with some purpose in mind like making further analysis easier, or performing filters.

==''k''-d tree==

A [http://en.wikipedia.org/wiki/K-d_tree ''k''-d tree] (''k''-dimensional tree) is a data structure that organizes a set of points in a ''k''-dimensional space, in a way that makes range search operations very efficient (for example, finding the nearest neighbor of a point, which is the point that is closest to it in space; or finding all neighbors within a radius).

It is a binary tree, that is, every non-leaf node in it has two children nodes, the "left" and "right" ones. Each level splits space on a specific dimension. For example, in 3-dimensional space, at the root node (first level) all children would be split based on the first dimension, X (points having coordinates with greater X values would go to the right subtree, points with lesser values to the left one). At the second level (the nodes we just created), the split would be done on the Y axis, following the same criteria. At the third level (the grandchildren), we would use the Z axis. At the fourth level, we would get back to the X axis, and so on. Usually, the median point is chosen to be root at every level.

[[Image:Kdtree.png|thumb|center|700px|Example of a 2D ''k''d-tree (image from Wikimedia Commons).]]

In PCL, creating a ''k''-d tree from a cloud is a straightforward and fast operation:

===Searching===

The following code snippet shows you how to use a ''k''-d tree to perform efficient searches:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/search/kdtree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// kd-tree object.
pcl::search::KdTree<pcl::PointXYZ> kdtree;
kdtree.setInputCloud(cloud);

// We will find the 5 nearest neighbors of this point
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices(5);
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances(5);
// Perform the search, and print out results.
if (kdtree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
if (kdtree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

Another alternative would be <span style="color:#FF1493">"KdTreeFLANN"</span>, which makes use of [http://www.cs.ubc.ca/research/flann/ FLANN] (Fast Library for Approximate Nearest Neighbors).

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/kdtree_search.php How to use a KdTree to search]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_kd_tree.html pcl::search::KdTree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1_kd_tree_f_l_a_n_n.html pcl::KdTreeFLANN]
* [http://robotica.unileon.es/~victorm/PCL_kdtree_search.tar.gz Download]
</div>

==Octree==

Like the ''k''-d tree, the [http://en.wikipedia.org/wiki/Octree octree] is an hierarchical tree data structure useful for searches, but also compression or downsampling. Each octree node (called a ''voxel'', a term that will sound familiar for users of 3D engines, and which could be described as a "3D pixel", that is, a cube) has either eight children or no children. The root node describes a cubic bounding box which encapsulates all points. At every level, it becomes subdivided by a factor of 2 which results in an increased
voxel resolution. That way, we can selectively give more resolution to certain zones.

[[Image:Octree.png|thumb|center|700px|Depiction of an octree (image from Wikimedia Commons).]]

===Searching===

Downsampling will be covered in a different section, so now I will show you how to perform searches with an octree:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>
#include <vector>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree object, with a maximum resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudSearch<pcl::PointXYZ> octree(128);
octree.setInputCloud(cloud);
octree.addPointsFromInputCloud();

// We will find all neighbors within the same voxel as this point.
// (it does not have to be one of the cloud's, we can use any coordinate).
pcl::PointXYZ point;
point.x = 0.0524343;
point.y = -0.58016;
point.z = 1.776;
// This vector will store the output neighbors.
std::vector<int> pointIndices;
// Perform the search, and print out results.
if (! octree.voxelSearch(point, pointIndices) > 0)
{
std::cout << "Neighbors in the same voxel:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z << std::endl;
}

// We will find the 5 nearest neighbors of the point.
// This vector will store their squared distances to the search point.
std::vector<float> squaredDistances;
if (octree.nearestKSearch(point, 5, pointIndices, squaredDistances) > 0)
{
std::cout << "5 nearest neighbors of the point:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}

// Now we find all neighbors within 3cm of the point
// (inside a sphere of radius 3cm centered at the point).
// The point DOES have to belong in the cloud.
if (octree.radiusSearch(point, 0.03, pointIndices, squaredDistances) > 0)
{
std::cout << "Neighbors within 3cm:" << std::endl;
for (size_t i = 0; i < pointIndices.size(); ++i)
std::cout << "\t" << cloud->points[pointIndices[i]].x
<< " " << cloud->points[pointIndices[i]].y
<< " " << cloud->points[pointIndices[i]].z
<< " (squared distance: " << squaredDistances[i] << ")" << std::endl;
}
}</syntaxhighlight>

As you can see, the code is mostly interchangeable with the one of the ''k''d-tree.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius | Neighbor count
* '''Output''': Point indices, Squared distances
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree.php Spatial Partitioning and Search Operations with Octrees]
* '''API''':
** [http://docs.pointclouds.org/trunk/classpcl_1_1search_1_1_octree.html pcl::search::Octree]
** [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_search.html pcl::octree::OctreePointCloudSearch]
* [http://robotica.unileon.es/~victorm/PCL_octree_search.tar.gz Download]
</div>

===Compressing/Decompressing===

On the other hand, this is the code needed to compress and decompress a point cloud with PCL:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/compression/octree_pointcloud_compression.h>
#include <boost/thread/thread.hpp>
#include <pcl/visualization/pcl_visualizer.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr decompressedCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Octree compressor object.
// Check /usr/include/pcl-<version>/pcl/compression/compression_profiles.h for more profiles.
// The second parameter enables the output of information.
pcl::io::OctreePointCloudCompression<pcl::PointXYZ> octreeCompression(pcl::io::MED_RES_ONLINE_COMPRESSION_WITHOUT_COLOR, true);
// Stringstream that will hold the compressed cloud.
std::stringstream compressedData;

// Compress the cloud (you would save the stream to disk).
octreeCompression.encodePointCloud(cloud, compressedData);

// Decompress the cloud.
octreeCompression.decodePointCloud(compressedData, decompressedCloud);

// Display the decompressed cloud.
boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("Octree compression"));
viewer->addPointCloud<pcl::PointXYZ>(decompressedCloud, "cloud");
while (!viewer->wasStopped())
{
viewer->spinOnce(100);
boost::this_thread::sleep(boost::posix_time::microseconds(100000));
}
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeCompression.png | Original point cloud.
File:AfterCompression.png | Point cloud retrieved after octree compression.
</gallery></center>

As you can see, the process has decreased the amount of points in the cloud. The default parameters of the <span style="color:#FF1493">"OctreePointCloudCompression"</span> object establish an octree resolution (at lowest level) of 1cm, and a coordinate precision of 1mm. These values can of course be changed to suit your needs, but simplifying the cloud in this way is an useful operation that we will cover later.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Resolution, Compression profile
* '''Output''': Compressed stream
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/compression.php Point Cloud Compression]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1io_1_1_octree_point_cloud_compression.html pcl::io::OctreePointCloudCompression]
* [http://robotica.unileon.es/~victorm/PCL_octree_compress.tar.gz Download]
</div>

===Detecting changes===

PCL has an octree-based object that allows to detect changes between two point clouds (for example, two clouds taken one after another with an OpenNI device). This object stores the trees of both clouds at the same time, and compares them, returning a list of indices of the points that are new in the second cloud (they did not exist in the first):

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/octree/octree.h>

#include <iostream>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudA(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloudB(new pcl::PointCloud<pcl::PointXYZ>);

// Read two PCD files from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloudA) != 0)
{
return -1;
}
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[2], *cloudB) != 0)
{
return -1;
}

// Change detector object, with a resolution of 128
// (resolution at lowest octree level).
pcl::octree::OctreePointCloudChangeDetector<pcl::PointXYZ> octree(128.0f);

// Add cloudA to the octree.
octree.setInputCloud(cloudA);
octree.addPointsFromInputCloud();
// The change detector object is able to store two clouds at the same time;
// with this we can reset the buffer but keep the previous tree saved.
octree.switchBuffers();
// Now, add cloudB to the octree like we did before.
octree.setInputCloud(cloudB);
octree.addPointsFromInputCloud();

std::vector<int> newPoints;
// Get a vector with the indices of all points that are new in cloud B,
// when compared with the ones that existed in cloud A.
octree.getPointIndicesFromNewVoxels(newPoints);
for (size_t i = 0; i < newPoints.size(); ++i)
{
std::cout << "Point (" << cloudB->points[newPoints[i]].x << ", "
<< cloudB->points[newPoints[i]].y << ", "
<< cloudB->points[newPoints[i]].z
<< ") was not in cloud A but is in cloud B" << std::endl;
}
std::cout << newPoints.size() << std::endl;
} </syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': First cloud, Second cloud, Resolution
* '''Output''': Point indices
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/octree_change.php Spatial change detection on unorganized point cloud data]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1octree_1_1_octree_point_cloud_change_detector.html pcl::octree::OctreePointCloudChangeDetector]
* [http://robotica.unileon.es/~victorm/PCL_change_detection.tar.gz Download]
</div>

=Filtering=

The raw point cloud, as returned from the sensor, usually needs some kind of polishing before being handed to processing algorithms. Normally, this is due to the low precision of the sensor, which leads to noise, incorrect measurements, holes in surfaces... Sometimes it is just the opposite: the cloud has so many points that processing it would take forever. Luckily, several filtering methods exist to "fix" many recurring errors in point clouds.

==Passthrough filter==

A passthrough filter will remove any point from the cloud whose values do not fall in a certain range, user-given. For example, if you wanted to discard all points farther away than, say, 3 meters, you would run the filter on the Z coordinate with a range of [0,3]. This filter can be useful to discard unneeded objects from the cloud, but you may have to adapt a different reference frame if the default one (relative to the sensor) is not fitting. For example, filtering on the Y value to remove all points not sitting on a table will yield unwanted results if the camera was at an odd angle.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/passthrough.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::PassThrough<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Filter out all points with Z values not in the [0-2] range.
filter.setFilterFieldName("z");
filter.setFilterLimits(0.0, 2.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforePassthrough.png | Original point cloud.
File:AfterPassthrough.png | Point cloud after applying a passthrough filter with Z in [0,2].
</gallery></center>

The <span style="color:#FF1493">"setFilterLimitsNegative()"</span> function (not seen in the example) will tell the filter object whether to return the points inside the range ("false", the default value) or outside it ("true").

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Filter field, Filter limits, [Negative mode]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/passthrough.php Filtering a PointCloud using a PassThrough filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_pass_through.html pcl::PassThrough]
* [http://robotica.unileon.es/~victorm/PCL_passthrough_filter.tar.gz Download]
</div>

==Conditional removal==

Another way of getting the exact same result than with the previous example would be to use a conditional removal filter. We can build any kind of condition we want for the values of the point:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/conditional_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// We must build a condition.
// And "And" condition requires all tests to check true. "Or" conditions also available.
pcl::ConditionAnd<pcl::PointXYZ>::Ptr condition(new pcl::ConditionAnd<pcl::PointXYZ>);
// First test, the point's Z value must be greater than (GT) 0.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::GT, 0.0)));
// Second test, the point's Z value must be less than (LT) 2.
condition->addComparison(pcl::FieldComparison<pcl::PointXYZ>::ConstPtr(new pcl::FieldComparison<pcl::PointXYZ>("z", pcl::ComparisonOps::LT, 2.0)));
// Checks available: GT, GE, LT, LE, EQ.

// Filter object.
pcl::ConditionalRemoval<pcl::PointXYZ> filter;
filter.setCondition(condition);
filter.setInputCloud(cloud);
// If true, points that do not pass the filter will be set to a certain value (default NaN).
// If false, they will be just removed, but that could break the structure of the cloud
// (organized clouds are clouds taken from camera-like sensors that return a matrix-like image).
filter.setKeepOrganized(true);
// If keep organized was set true, points that failed the test will have their Z value set to this.
filter.setUserFilterValue(0.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Condition, [Keep organized]
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_conditional_removal.html pcl::ConditionalRemoval]
* [http://robotica.unileon.es/~victorm/PCL_conditional_removal.tar.gz Download]
</div>

==Outlier removal==

Outliers are lonely points that are spread here and there in the cloud, like annoying mosquitoes. They are the product of the sensor's inaccuracy, which registers measurements where there should not be any. Outliers are considered undesired noise because they can introduce errors in calculations, like for example the normal estimation. Hence, trimming they out of the cloud will not only make computations faster (because the less points there are, the less time it takes, as we will see with downsampling) but also help us get more precise values.

===Radius-based===

The radius-based outlier removal is the simplest method of all. You must specify a search radius and the minimum number of neighbors than a point must have to avoid being labelled as outlier. The algorithm will then iterate through all points (which can be slow in if the cloud is big) and perform the check: if less than that number of points are found within the radius, it is removed.

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/radius_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::RadiusOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Every point must have 10 neighbors within 15cm, or it will be removed.
filter.setRadiusSearch(0.15);
filter.setMinNeighborsInRadius(10);

filter.filter(*filteredCloud);
}</syntaxhighlight>

All PCL classes that inherit from <span style="color:#FF1493">"pcl::FilterIndices"</span> (like <span style="color:#FF1493">"pcl::RadiusOutlierRemoval"</span>) provide the function <span style="color:#FF1493">"setNegative()"</span>, which if set to true, makes the filter return the points that did NOT pass the filter.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Radius, Neighbor count
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/remove_outliers.php Removing outliers using a Conditional or RadiusOutlier removal]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_radius_outlier_removal.html pcl::RadiusOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_radius_outlier_removal.tar.gz Download]
</div>

===Statistical===

The statistical outlier removal process is a bit more refined. First, for every point, the mean distance to its ''K'' neighbors is computed. Then, if we asume that the result is a [http://en.wikipedia.org/wiki/Normal_distribution normal (gaussian) distribution] with a mean ''μ'' and a standard deviation ''σ'', we can deem it safe to remove all points with mean distances that fall out of the global mean plus deviation. Basically, it runs a statistical analysis of the distances between neighboring points, and trims all which are not considered "normal" (you define what "normal" is with the parameters of the algorithm).

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/statistical_outlier_removal.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::StatisticalOutlierRemoval<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Set number of neighbors to consider to 50.
filter.setMeanK(50);
// Set standard deviation multiplier to 1.
// Points with a distance larger than 1 standard deviation of the mean distance will be outliers.
filter.setStddevMulThresh(1.0);

filter.filter(*filteredCloud);
}</syntaxhighlight>

You can make use of <span style="color:#FF1493">"setNegative()"</span> with this filter too.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Neighbor count, Deviation multiplier
* '''Output''': Filtered points
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/statistical_outlier.php Removing outliers using a StatisticalOutlierRemoval filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_statistical_outlier_removal.html pcl::StatisticalOutlierRemoval]
* [http://robotica.unileon.es/~victorm/PCL_statistical_outlier_removal.tar.gz Download]
</div>

==Resampling==

Resampling means changing the number of points of the cloud, either by reducing (downsampling) or increasing (upsampling). Both have a purpose, as we will see.

===Downsampling===

A sensor like Kinect or Xtion produces a cloud with 307200 (640x480) points. Believe it or not, that is a lot of information, and the new Kinect is already out there, working at a higher resolution. Keep in mind that, if we were to perform a simple operation on every point of a cloud, it would be of ''O(n)'', n being the number of points. If we had to compare every point with its ''k'' nearest neighbors, it would be ''O(nk)''. You may have noticed by now that some of the algorithms I showed you take several seconds to run on your PC; you now know why.

Optimization techniques are being implemented as I write this to allow PCL to employ the computer's GPU (graphic card) to perform the computations using [https://developer.nvidia.com/about-cuda CUDA]. Many classes already provide an alternative in the <span style="color:#FF1493">"pcl::gpu"</span> namespace. This offers dramatic improvements in speed, but it is not always a possibility.

Another option is to reduce the complexity of the cloud, that is, to remove all points that are not needed for the purpose. There are several ways to do this. You could trim all outliers from the cloud, then perform segmentation to extract only the parts that interest you. A common operation is also downsampling, which will give you back a cloud that, for all intents and purposes, equivalent to the original one, despite having less points.

Downsampling is done via ''voxelization''. I already explained what voxels were in the [[#Octree | octree section]]. The cloud is divided in multiple cube-shaped regions with the desired resolution. Then, all points inside every voxel are processed so only one remains. The simplest way would be to randomly select one of them, but a more accurate approach would be to compute the [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Computing_the_centroid|centroid]], which is the point whose coordinates are the mean values of all the points that belong to the voxel.

If done with sensible parameters, downsampling will yield results that are precise enough to work with, at the cost of less CPU power.

====Voxel grid====

A straightforward way of downsampling a cloud:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filter object.
pcl::VoxelGrid<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setLeafSize(0.01f, 0.01f, 0.01f);

filter.filter(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:BeforeDownsampling.png | Original point cloud.
File:AfterDownsampling.png | Point cloud after being downsampled with a leaf size of 1cm.
</gallery></center>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Downsampled cloud
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/voxel_grid.php Downsampling a PointCloud using a VoxelGrid filter]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_voxel_grid.html pcl::VoxelGrid]
* [http://robotica.unileon.es/~victorm/PCL_downsample.tar.gz Download]
</div>

====Uniform sampling====

This class does basically the same, but it outputs the indices of the points that survived the process. It is a common way of choosing keypoints when computing [[PCL/OpenNI_tutorial_4:_3D_object_recognition_%28descriptors%29|descriptors]].

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/keypoints/uniform_sampling.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Uniform sampling object.
pcl::UniformSampling<pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// We set the size of every voxel to be 1x1x1cm
// (only one point per every cubic centimeter will survive).
filter.setRadiusSearch(0.01f);
// We need an additional object to store the indices of surviving points.
pcl::PointCloud<int> keypointIndices;

filter.compute(keypointIndices);
pcl::copyPointCloud(*cloud, keypointIndices.points, *filteredCloud);
}</syntaxhighlight>

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Voxel size
* '''Output''': Point indices
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_uniform_sampling.html pcl::UniformSampling]
* [http://robotica.unileon.es/~victorm/PCL_downsample_uniform.tar.gz Download]
</div>

===Upsampling===

Upsampling is a form of [[PCL/OpenNI_tutorial_2:_Cloud_processing_%28basic%29#Reconstruction| surface reconstruction]], but I put it here. When you have fewer points than you would like, upsampling can help you to recover the original surface(s), by [http://en.wikipedia.org/wiki/Upsampling interpolating] the ones you currently have, which is just a sophisticated guess. The result will never be a hundred percent accurate, but sometimes it is the only choice. So, before downsampling your clouds, be sure to keep a copy of the original data!

PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares Moving Least Squares (MLS)] algorithm for this. The code looks like:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Objects for storing the point clouds.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr filteredCloud(new pcl::PointCloud<pcl::PointXYZ>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Filtering object.
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ> filter;
filter.setInputCloud(cloud);
// Object for searching.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// Upsampling method. Other possibilites are DISTINCT_CLOUD, RANDOM_UNIFORM_DENSITY
// and VOXEL_GRID_DILATION. NONE disables upsampling. Check the API for details.
filter.setUpsamplingMethod(pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointXYZ>::SAMPLE_LOCAL_PLANE);
// Radius around each point, where the local plane will be sampled.
filter.setUpsamplingRadius(0.03);
// Sampling step size. Bigger values will yield less (if any) new points.
filter.setUpsamplingStepSize(0.02);

filter.process(*filteredCloud);
}</syntaxhighlight>

<center><gallery widths=300px>
File:PCL_upsampling_original.png | Original point cloud (coffee mug on a table).
File:PCL_upsampling_downsampled.png | Downsampled with a leaf size of 1cm.
File:PCL_upsampling_local_plane.png | Upsampled using SAMPLE_LOCAL_PLANE (upsampling radius 3cm, step size 2cm).
File:PCL_upsampling_uniform_density.png | Upsampled using RANDOM_UNIFORM_DENSITY (target point density of 300 per 3cm radius).
</gallery></center>

Check the [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html#ac29ad97b98353d64ce64e2ff924f7d20 setUpsamplingMethod()] API to see all upsampling methods available, and the parameters you need to set for them to work.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, Upsampling method, [Upsampling parameters]
* '''Output''': Upsampled cloud
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_upsample.tar.gz Download]
</div>

=Reconstruction=

==Surface smoothing==

As stated, depth sensors are not very accurate, and the resulting clouds have measurement errors, outliers, holes in surfaces, etc. Surfaces can be reconstructed by means of an algorithm, that iterates through all points and interpolates the data, trying to guess how the original surface was. Like with upsampling, PCL uses the [http://en.wikipedia.org/wiki/Moving_least_squares MLS] algorithm and class. Performing this step is important, because the resulting cloud's normals will be more accurate.

Surface smoothing is easily done with the following code:

<syntaxhighlight lang=CPP>#include <pcl/io/pcd_io.h>
#include <pcl/surface/mls.h>

int
main(int argc, char** argv)
{
// Object for storing the point cloud.
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
// The output will also contain the normals.
pcl::PointCloud<pcl::PointNormal>::Ptr smoothedCloud(new pcl::PointCloud<pcl::PointNormal>);

// Read a PCD file from disk.
if (pcl::io::loadPCDFile<pcl::PointXYZ>(argv[1], *cloud) != 0)
{
return -1;
}

// Smoothing object (we choose what point types we want as input and output).
pcl::MovingLeastSquares<pcl::PointXYZ, pcl::PointNormal> filter;
filter.setInputCloud(cloud);
// Use all neighbors in a radius of 3cm.
filter.setSearchRadius(0.03);
// If true, the surface and normal are approximated using a polynomial estimation
// (if false, only a tangent one).
filter.setPolynomialFit(true);
// We can tell the algorithm to also compute smoothed normals (optional).
filter.setComputeNormals(true);
// kd-tree object for performing searches.
pcl::search::KdTree<pcl::PointXYZ>::Ptr kdtree;
filter.setSearchMethod(kdtree);

filter.process(*smoothedCloud);
}</syntaxhighlight>

The output cloud will also contain the improved normals of the smoother cloud.

<div style="background-color: #F8F8F8; border-style: dotted;">
* '''Input''': Points, Search radius, [Polynomial fit], [Compute normals]
* '''Output''': Smoothed cloud, [Smoothed normals]
* '''Tutorial''': [http://pointclouds.org/documentation/tutorials/resampling.php Smoothing and normal estimation based on polynomial reconstruction]
* '''API''': [http://docs.pointclouds.org/trunk/classpcl_1_1_moving_least_squares.html pcl::MovingLeastSquares]
* [http://robotica.unileon.es/~victorm/PCL_surface_smoothing.tar.gz Download]
</div>

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 1: Installing and testing

2015-11-01T22:59:22Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Kinect.jpg|thumb|right|200px|Microsoft Kinect device.]]

This series of tutorials will explain the usage of a depth camera like [https://en.wikipedia.org/wiki/Kinect Kinect] for "serious" researching purposes. As you may know, Kinect is in fact an affordable depth sensor, developed with technology from [https://en.wikipedia.org/wiki/PrimeSense PrimeSense], based on infrarred structured light method. It also has a common camera (which makes it a RGB-D device), a microphone and a motorized pivot. Its use is not limited to playing with a Xbox360 console, you can plug it to a computer and use it like any other sensor. Many open-source drivers and frameworks are available.

Since its release on November 2010, it has gained a lot of popularity, specially among the scientific community. Many researches have procured themselves one because, despite the low cost (about 100 $), it has proven to be a powerful solution for depth sensing projects. Current investigations focus on real-time surface mapping, object recognition and tracking, and localization. Impressive results (like the [http://research.microsoft.com/en-us/projects/surfacerecon/ KinectFusion] project from Microsoft) are already possible.

The new [https://en.wikipedia.org/wiki/Xbox_One Xbox One] ships with an upgraded version, [https://en.wikipedia.org/wiki/Kinect_for_Xbox_One Kinect v2], with enhanced resolution, that is able to detect your facial expression, measure your heart rate, and track every one of your fingers. A PC development-ready version ([https://en.wikipedia.org/wiki/Kinect_for_Xbox_One#Kinect_for_Windows Kinect for Windows v2]) was released in July 2014, but it could only be used with the official Windows SDK (open source support [https://github.com/OpenKinect/libfreenect2 exists] but is still young). In October 2014 an adapter that allows to plug the standard Kinect v2 to a PC was released, so the development version of the sensor was discontinued in April 2015. Now you can just buy the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-for-Xbox-One/productID.307499400 standard one] for the console plus the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600 adapter].

I will explain the installation and usage of one of the "legacy" Kinect 1.0 devices with a common PC, and the possibilities it offers. I will do it in an easy to understand way, intended for students that have just acquired it and want to start from scratch.

Keep in mind that the software that we are going to use (Point Cloud Library and OpenNI drivers) will also let you use any other device like the [https://www.asus.com/3D-Sensor/Xtion_PRO/ Xtion PRO] or [https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/ Xtion PRO LIVE] from ASUS (the PRO only has a depth sensor, the PRO LIVE is a RGB-D camera) without changing a line of code.

<span style="color:#606060">'''''NOTE: The tutorials are written for Linux platforms. Also, 64-bit versions seem to work better than 32-bit.'''''</span>

=Requirements=

You will need the following:

* A common Kinect device, out of the box. You can buy it in your local electronics shop, or online. It also includes a free copy of ''[https://en.wikipedia.org/wiki/Kinect_Adventures! Kinect Adventures!]'', which is useless if you do not own the console. Microsoft has released a ''Kinect for Windows'' device, which is a normal looking Kinect no longer compatible with Xbox360, that will only work with their official SDK, intended for developers only. Also, like I stated earlier, you can use an ASUS Xtion indistinctly.
* A computer running Linux (Debian or Ubuntu preferably).
* A medium-sized room. Kinect has some limitations for depth measurement: 40cm minimum, 8m maximum (make it 6).

<span style="color:#606060">'''''NOTE: ''Kinect for Windows'' may have problems working with open source drivers on Linux .'''''</span>

=Connecting everything=

Kinect does not work with a common USB port. Its power consumption is a bit higher because of the motor, so Microsoft came up with a connector that combines USB and power supply. Old Xbox 360 models needed a special adapter, new S model ones already have this new port. Luckily, Kinect comes with the official adapter out of the box (otherwise you will have to [http://www.amazon.com/Power-Supply-Cable-Kinect-Xbox-360/dp/B004S7GA46/ref=sr_1_1?ie=UTF8&qid=1378288327&sr=8-1 buy one]).

Just plug the adapter to any power socket, and the USB to your computer. Let's check typing this in a terminal:

<syntaxhighlight lang=Bash>lsusb</syntaxhighlight>

Output should list the following devices:

<syntaxhighlight lang=Bash>Bus 001 Device 005: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 006: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 007: ID 045e:02ae Microsoft Corp. Xbox NUI Camera</syntaxhighlight>

If you are using a Xtion, you should see:

<syntaxhighlight lang=Bash>Bus 001 Device 004: ID 1d27:0601 ASUS</syntaxhighlight>

"0601" identifies the new Xtion model, "0600" the older one. Both should work the same. But try to avoid [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO|USB 3.0 ports]]!

=Installing the software=

There is more than one way to get your Kinect working on your PC, and start developing applications for it:

* [https://www.microsoft.com/en-us/download/details.aspx?id=40278 Kinect for Windows SDK] and [https://www.microsoft.com/en-us/download/details.aspx?id=40276 Developer Toolkit]: released on June 16, 2011 as a non-commercial SDK intended for application development. Version 1.8, the last there will ever be now that Kinect v2 is out, was released on September 2013. Because it comes from Microsoft, it is obviously the easiest way to get everything working. Sadly, there is no Linux version.
* [https://github.com/OpenKinect/libfreenect libfreenect] library: from the [http://openkinect.org/wiki/Main_Page OpenKinect] project, it is intended to be a free and open source alternative to the official drivers. libfreenect is used by projects like [https://github.com/ofTheo/ofxKinect ofxKinect], an addon for the [http://www.openframeworks.cc/ openFrameworks] toolkit (and as of version 0.8, included in the core package) that runs on Linux and OS X. ofxKinect packs a nice example application to show the RGB and point cloud taken from Kinect.
* [https://github.com/PrimeSense/Sensor PrimeSense drivers]: they were released as open source after the the [https://en.wikipedia.org/wiki/OpenNI OpenNI] project was created, along with the motion tracking middleware, ''NITE'', and the SDK. NI stands for Natural Interaction, and the project tried to enforce a common standard for human input using Kinect-like sensors. These official drivers are used by [http://wiki.ros.org/openni_kinect ROS] (the Robot Operating System, a massive collection of libraries and tools for robotic researchers) and [http://pointclouds.org/ PCL] (the Point Cloud Library, with everything needed for 3D point cloud processing). Sadly, version 2.0 of the OpenNI SDK dropped support for Kinect on Linux due to licensing issues, so for now PCL relies on legacy 1.x versions. Also, Apple bought PrimeSense on November 2013, and on April 2014 OpenNI's webpage was closed. The source is now being maintained by [http://structure.io/openni a third party].
* [https://github.com/avin2/SensorKinect SensorKinect]: a modified version of the official PrimeSense drivers, used for example by [https://github.com/gameoverhack/ofxOpenNI ofxOpenNI] (another openFrameworks addon). Last updated on 2012.

For this tutorial, we are going to use PCL with the OpenNI drivers, so owners of a Xtion can also get it to work.

==Precompiled PCL for Ubuntu==

If you have a valid [http://wiki.ros.org/ROS/Installation installation of ROS] (through their repository), you do not have to install anything. Both OpenNI and PrimeSense drivers, as well as PCL, should be already installed. You can check it with:

<syntaxhighlight lang=Bash>sudo apt-get install libpcl-1.7-all libpcl-1.7-all-dev libopenni-dev libopenni-sensor-primesense-dev</syntaxhighlight>

The previous command should state that all packages are already installed (change the PCL version number as needed), install them if not. If you get an error about overwriting some file, check [[PCL/OpenNI_troubleshooting#Trying_to_overwrite_X.2C_which_is_also_in_package_Y|this]].

In case you do not want ROS, there is a [https://launchpad.net/~v-launchpad-jochen-sprickerhof-de/+archive/ubuntu/pcl PPA] (Personal Package Archive, a private repository) which has everything we need. Add it to your sources, and install everything:

<syntaxhighlight lang=Bash>sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install build-essential cmake libpcl1.7 libpcl-dev pcl-tools</syntaxhighlight>

Trying to mix ROS and PCL repositories and packages can cause some errors, so choose one of them and stick with it. Check the [[PCL/OpenNI troubleshooting]] page because your Kinect may not work by default in 32 bits.

==Compiling PCL from source==

For Linuxes without a precompiled version of PCL, you will need to compile it yourself. This has an advantage, actually: you can customize the build options and choose what you want. Also, the resulting binaries and libraries should be a bit faster. And you always get the latest version! The instructions are [http://pointclouds.org/documentation/tutorials/compiling_pcl_posix.php here], but the steps are easy so I will show them to you.

===Installing the dependencies===

Some of PCL dependencies can be installed via the package manager. Others will require additional work.

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install build-essential cmake cmake-curses-gui libboost-all-dev libeigen3-dev libflann-dev libvtk5-dev libvtk5-qt4-dev libglew-dev libxmu-dev libsuitesparse-dev libqhull-dev libpcap-dev libxmu-dev libxi-dev libgtest-dev libqt4-opengl-dev</syntaxhighlight>

The trunk version of PCL uses [https://en.wikipedia.org/wiki/VTK VTK] 6 and [https://en.wikipedia.org/wiki/Qt_%28software%29 Qt] 5, so if you intend to compile it, you must install the following packages (say yes if you are asked to remove VTK 5 and Qt 4 packages):

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

====JDK====

OpenNI requires Sun's official JDK (Java Development Kit), which is no longer available on official apt repositories. You can use [http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html unofficial ones], or do it manually. For the last method, go to the [http://www.oracle.com/technetwork/java/javase/downloads/index.html Java SE downloads] page (SE means Standard Edition) and download the latest version (e.g., <span style="color:#1E90FF">''jdk-8u66-linux-x64.tar.gz''</span>). Extract it, then move the contents to <span style="color:#FF8C00">''/usr/lib/jvm/''</span> so it is available system-wide:

<syntaxhighlight lang=Bash>sudo mkdir -p /usr/lib/jvm/
sudo cp -r jdk1.8.0_66/ /usr/lib/jvm/</syntaxhighlight>

Then, make it the default choice to compile and run Java programs. Remember to change the version number as needed!

<syntaxhighlight lang=Bash>sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_66/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_66/bin/javac" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.8.0_66/bin/jar" 1</syntaxhighlight>

To be sure, use the following commands to manually select the correct option, in case there is more than one choice:

<syntaxhighlight lang=Bash>sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config jar</syntaxhighlight>

If you are still not sure, run them and display the version, making sure it is the one you installed:

<syntaxhighlight lang=Bash>java -version
javac -version</syntaxhighlight>

Sun's JDK is now installed.

====OpenNI====

PCL uses OpenNI and the PrimeSense drivers to get data from devices like Kinect or Xtion. It is optional, but it would not make much sense not to install it, would it? If you are using Ubuntu, add the [[#Precompiled PCL for Ubuntu | PPA]] and install <span style="color:#228B22">''libopenni-dev''</span> and <span style="color:#228B22">''libopenni-sensor-primesense-dev''</span>, which should be done already. Otherwise, fetch the [https://github.com/OpenNI/OpenNI OpenNI] and [https://github.com/PrimeSense/Sensor PrimeSense Sensor] sources from GitHub (download them as .zip, the link is on the right). Extract them, and install the dependencies:

<syntaxhighlight lang=Bash>sudo apt-get install python libusb-1.0-0-dev freeglut3-dev doxygen graphviz</syntaxhighlight>

Go to the directory where you extracted OpenNI (<span style="color:#FF8C00">''OpenNI-master/''</span> for me), and open a terminal in the <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span> subdirectory. Issue:

<syntaxhighlight lang=Bash>./RedistMaker</syntaxhighlight>

When it finishes, and if there are no errors (check the [[PCL/OpenNI troubleshooting]] page if you get any), go to <span style="color:#FF8C00">''Platform/Linux/Redist/OpenNI-Bin-Dev-Linux-x64-v1.5.7.10/''</span> (or your equivalent), and install (you must be root):

<syntaxhighlight lang=Bash>sudo ./install.sh</syntaxhighlight>

Now, go to the directory where you extracted the PrimeSense drivers (<span style="color:#FF8C00">''Sensor-master/''</span> for me), and repeat the exact same procedure (go to <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span>, issue <span style="color:#228B22">''./RedistMaker''</span>, go to <span style="color:#FF8C00">''Platform/Linux/Redist/Sensor-Bin-Linux-x64-v5.1.6.6/''</span>, issue <span style="color:#228B22">''sudo ./install.sh''</span>). Congratulations, you have now installed OpenNI.

====CUDA====

Like OpenNI, [https://en.wikipedia.org/wiki/CUDA nVidia CUDA] is an optional dependency, that will allow PCL to use your GPU (Graphics Processing Unit, that is, your graphics card) for certain computations. This is mandatory for tools like KinFu (do not bother unless you have at least a series 400 card with 1.5 GB of VRAM).

Some distributions provide packages for CUDA in the repositories. For example, in Ubuntu:

<syntaxhighlight lang=Bash>sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit</syntaxhighlight>

If you want to install it manually (which is incompatible with the previous method), go to the [https://developer.nvidia.com/cuda-downloads CUDA downloads] page, which is self-explanatory, and get the small .deb file or the huge .run toolkit/SDK installer file for your system (you should have installed the nVidia drivers already, but the installer will also do it for you if needed).

If you chose the .deb file, install it using the method of your choice, like Gdebi package manager, or through console:

<syntaxhighlight lang=Bash>sudo dpkg -i <package.deb></syntaxhighlight>

The .deb does not contain all CUDA stuff, it just adds their repository to your software lists. Now you must install everything:

<syntaxhighlight lang=Bash>sudo apt-get update
sudo apt-get install cuda</syntaxhighlight>

If, on the other hand, you downloaded the .run, give it execution permissions:

<syntaxhighlight lang=Bash>chmod +x cuda_7.0.28_linux.run</syntaxhighlight>

And install it. You can use the default options, although if you have a working nVidia graphics driver for your card, you may want to say "no" when the installer offers to install it for you:

<syntaxhighlight lang=Bash>sudo ./cuda_7.0.28_linux.run</syntaxhighlight>

The environment variables need to be changed so your system can find CUDA's libraries and binaries. Open <span style="color:#1E90FF">''/etc/ld.so.conf''</span>:

<syntaxhighlight lang=Bash>sudo nano /etc/ld.so.conf</syntaxhighlight>

And append one of these two lines:

<syntaxhighlight lang=Bash>/usr/local/cuda/lib64 # Add this on 64-bit only.
/usr/local/cuda/lib # Add this on 32-bit only.</syntaxhighlight>

Save with <span style="color:#228B22">'''Ctrl+O'''</span> and Enter, exit with <span style="color:#228B22">'''Ctrl+X'''</span>. Reload the cache of the dynamic linker with:

<syntaxhighlight lang=Bash>sudo ldconfig</syntaxhighlight>

Now, append CUDA's bin directory to your PATH. Do this by editing your local <span style="color:#1E90FF">''.bashrc''</span> file:

<syntaxhighlight lang=Bash>nano ~/.bashrc</syntaxhighlight>

And append this line:

<syntaxhighlight lang=Bash>export PATH=$PATH:/usr/local/cuda/bin</syntaxhighlight>

CUDA is now installed.

===Getting the source===

Every dependency is installed. Time to download PCL's source code. First, you must choose whether to install the stable or the experimental branch of PCL. The stable branch is the latest official release and it is guaranteed to work without problems. The experimental branch may give you a compiling error seldomly, but you can find some interesting features that stable users will have to wait some months for. Apart from that, both are built the same way.

To get the stable version, go to the [https://github.com/PointCloudLibrary/pcl/releases downloads] page, get <span style="color:#1E90FF">''pcl-pcl-1.7.2.tar.gz''</span> or whatever the latest release is, and extract it somewhere. For the experimental version, use Git:

<syntaxhighlight lang=Bash>sudo apt-get install git
git clone https://github.com/PointCloudLibrary/pcl PCL-trunk-Source</syntaxhighlight>

The compiled trunk version of PCL will take up more than 8 GB. So make sure you put the source folder in a partition with enough free space!

===Compiling===

Go the the PCL source directory (<span style="color:#FF8C00">''pcl-pcl-1.7.2/''</span> or <span style="color:#FF8C00">''PCL-trunk-Source/''</span>), and create a new subdirectory to keep the build files in:

<syntaxhighlight lang=Bash>mkdir build
cd build</syntaxhighlight>

Now it is time to configure the project using CMake. We will tell it to build in Release (fully optimized, no debug capabilities) mode now, and customize the rest of the options later:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..</syntaxhighlight>

CMake should be able to find every dependency, thus being able to build every subsystem except for the ones marked as <span style="color:#FF1493">''"Disabled by default"''</span>. If you are happy, you can build now, otherwise let's invoke CMake's curses interface to change a couple of things (mind the final dot):

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

[[Image:Ccmake_GUI_PCL.png|thumb|center|900px|Interface of ''ccmake''.]]

Here you can change the build options. The program usage can be found at the bottom of the screen. Try turning all functionality <span style="color:#FF1493">"ON"</span>. The most important thing, in case you want to use CUDA, is to enable it and give CMake the path to your SDK. Go to the <span style="color:#FF1493">"CUDA_SDK_ROOT_DIR"</span> option and enter the correct path (probably <span style="color:#FF8C00">''/usr/local/cuda/''</span> or similar).

When you are done, press <span style="color:#228B22">'''C'''</span> to configure and <span style="color:#228B22">'''G'''</span> to generate and exit the tool. Sometimes, the options you change can activate previously omitted parameters, or prompt some warning text. Just press <span style="color:#228B22">'''E'''</span> when you are finished reading the message, and keep pressing <span style="color:#228B22">'''C'''</span> until it lets you generate (new parameters will be marked with an asterisk, so you can check them and decide whether or not you want further customization).

If you are done configuring, it is time to build:

<syntaxhighlight lang=Bash>make</syntaxhighlight>

<span style="color:#606060">'''''NOTE: Additionally, you can append the parameter -jX to speed up the compilation, X being the number of cores or processors of your PC, plus one.'''''</span>

Remember that, at any time, you can manually force the project to be reconfigured and built from scratch by emptying the <span style="color:#FF8C00">''build/''</span> directory with:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

===Installing===

It will take some time to compile PCL (up to a few hours if your PC is not powerful enough). When it is finished, install it system-wide with:

<syntaxhighlight lang=Bash>sudo make install</syntaxhighlight>

And you should reboot and proceed to the next section, to see if your computer now recognizes (and uses) your Kinect device.

=Testing (OpenNI viewer)=

We are going to write a simple example program that will fetch data from the Kinect or Xtion and present it to the user, using the PCL library. It will also allow to save the current frame (as point cloud) to disk. If you feel lazy, you can download it [http://robotica.unileon.es/~victorm/PCL_openni_viewer.tar.gz here], and skip the next two sections. Otherwise, create a new directory anywhere in your hard disk and proceed.

==CMakeLists.txt==

Inside that directory, create a new text file named <span style="color:#1E90FF">''CMakeLists.txt''</span>. PCL-based programs use the CMake build system, too. Open it with any editor and paste the following content:

<syntaxhighlight lang=CMake>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

project(PCL_openni_viewer)

find_package(PCL 1.7 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

set(PCL_BUILD_TYPE Release)

file(GLOB PCL_openni_viewer_SRC
"src/*.h"
"src/*.cpp"
)
add_executable(openniViewer ${PCL_openni_viewer_SRC})

target_link_libraries (openniViewer ${PCL_LIBRARIES})</syntaxhighlight>

CMake syntax is quite self-explanatory. We ask for a CMake version 2.8 installation, minimum. We declare a new project named <span style="color:#FF1493">"openni_PCL_viewer"</span>. We tell CMake to check for the presence of PCL library development files, version 1.7. If our system can not meet the CMake and PCL version requirement, the process will fail.

Next, we feed the compiler and linker the directories where PCL includes and libraries can be found, and the defined symbols. We tell CMake to use the <span style="color:#FF1493">"Release"</span> build type, which will activate certain optimizations depending on the compiler we use. Other build types are available, like <span style="color:#FF1493">"Debug"</span>, <span style="color:#FF1493">"MinSizeRel"</span>, and <span style="color:#FF1493">"RelWithDebInfo"</span>.

Finally, we create a variable, <span style="color:#FF1493">"opennipclviewer_SRC"</span>, that will store a list of files to be compiled (though we will only have one). We create a new binary to be compiled from these source files, and we link it with the PCL library.

Check the [https://cmake.org/Wiki/CMake_Useful_Variables CMake help] for more interesting options.

==main.cpp==

We told CMake it could find the source files in a <span style="color:#FF8C00">''src/''</span> subdirectory, so let's keep to out word and create it. Then, add a new <span style="color:#1E90FF">''main.cpp''</span> file inside and paste the following lines:

<syntaxhighlight lang=CPP>// Original code by Geoffrey Biggs, taken from the PCL tutorial in
// http://pointclouds.org/documentation/tutorials/pcl_visualizer.php

// Simple OpenNI viewer that also allows to write the current scene to a .pcd
// when pressing SPACE.

#include <pcl/io/openni_grabber.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/console/parse.h>

#include <iostream>

using namespace std;
using namespace pcl;

PointCloud<PointXYZRGBA>::Ptr cloudptr(new PointCloud<PointXYZRGBA>); // A cloud that will store color info.
PointCloud<PointXYZ>::Ptr fallbackCloud(new PointCloud<PointXYZ>); // A fallback cloud with just depth data.
boost::shared_ptr<visualization::CloudViewer> viewer; // Point cloud viewer object.
Grabber* openniGrabber; // OpenNI grabber that takes data from the device.
unsigned int filesSaved = 0; // For the numbering of the clouds saved to disk.
bool saveCloud(false), noColor(false); // Program control.

void
printUsage(const char* programName)
{
cout << "Usage: " << programName << " [options]"
<< endl
<< endl
<< "Options:\n"
<< endl
<< "\t<none> start capturing from an OpenNI device.\n"
<< "\t-v FILE visualize the given .pcd file.\n"
<< "\t-h shows this help.\n";
}

// This function is called every time the device has new data.
void
grabberCallback(const PointCloud<PointXYZRGBA>::ConstPtr& cloud)
{
if (! viewer->wasStopped())
viewer->showCloud(cloud);

if (saveCloud)
{
stringstream stream;
stream << "inputCloud" << filesSaved << ".pcd";
string filename = stream.str();
if (io::savePCDFile(filename, *cloud, true) == 0)
{
filesSaved++;
cout << "Saved " << filename << "." << endl;
}
else PCL_ERROR("Problem saving %s.\n", filename.c_str());

saveCloud = false;
}
}

// For detecting when SPACE is pressed.
void
keyboardEventOccurred(const visualization::KeyboardEvent& event,
void* nothing)
{
if (event.getKeySym() == "space" && event.keyDown())
saveCloud = true;
}

// Creates, initializes and returns a new viewer.
boost::shared_ptr<visualization::CloudViewer>
createViewer()
{
boost::shared_ptr<visualization::CloudViewer> v
(new visualization::CloudViewer("OpenNI viewer"));
v->registerKeyboardCallback(keyboardEventOccurred);

return (v);
}

int
main(int argc, char** argv)
{
if (console::find_argument(argc, argv, "-h") >= 0)
{
printUsage(argv[0]);
return -1;
}

bool justVisualize(false);
string filename;
if (console::find_argument(argc, argv, "-v") >= 0)
{
if (argc != 3)
{
printUsage(argv[0]);
return -1;
}

filename = argv[2];
justVisualize = true;
}
else if (argc != 1)
{
printUsage(argv[0]);
return -1;
}

// First mode, open and show a cloud from disk.
if (justVisualize)
{
// Try with color information...
try
{
io::loadPCDFile<PointXYZRGBA>(filename.c_str(), *cloudptr);
}
catch (PCLException e1)
{
try
{
// ...and if it fails, fall back to just depth.
io::loadPCDFile<PointXYZ>(filename.c_str(), *fallbackCloud);
}
catch (PCLException e2)
{
return -1;
}

noColor = true;
}

cout << "Loaded " << filename << "." << endl;
if (noColor)
cout << "This cloud has no RGBA color information present." << endl;
else cout << "This cloud has RGBA color information present." << endl;
}
// Second mode, start fetching and displaying frames from the device.
else
{
openniGrabber = new OpenNIGrabber();
if (openniGrabber == 0)
return -1;
boost::function<void (const PointCloud<PointXYZRGBA>::ConstPtr&)> f =
boost::bind(&grabberCallback, _1);
openniGrabber->registerCallback(f);
}

viewer = createViewer();

if (justVisualize)
{
if (noColor)
viewer->showCloud(fallbackCloud);
else viewer->showCloud(cloudptr);
}
else openniGrabber->start();

// Main loop.
while (! viewer->wasStopped())
boost::this_thread::sleep(boost::posix_time::seconds(1));

if (! justVisualize)
openniGrabber->stop();
}</syntaxhighlight>

Save and close.

==Compiling==

Follow the same steps you used to build PCL. That is, create a new <span style="color:#FF8C00">''build/''</span> subdirectory next to the <span style="color:#FF8C00">''src/''</span> one. Open a terminal there and issue:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..
make</syntaxhighlight>

==Executing==

Still from the same terminal, run the compiled example program:

<syntaxhighlight lang=Bash>./openniViewer</syntaxhighlight>

After some seconds, the main window will appear and the application will start grabbing frames from the device. You can inspect the current point cloud using the mouse, holding the left button to rotate, the right one (or the mouse wheel) to zoom, and the middle one to pan the camera around. At first, you may see only a black screen, or some big colored axes, but no cloud. Try zooming out, to see the whole scene. Another useful key is <span style="color:#228B22">'''R'''</span>, which will reset the camera parameters when pressed. Use it whenever you notice that zooming has gotten slow after some camera movement, or if you still can not see the cloud. See the [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer] tutorial for additional controls and features.

Whenever you feel ready, press the <span style="color:#228B22">'''SPACE'''</span> key. The program will pause for a fraction of a second and the output <span style="color:#FF1493">''"Saved inputCloud0.pcd."''</span> will appear on the console. Check the current folder to see that file <span style="color:#1E90FF">''inputCloud0.pcd''</span> has indeed been written. You can now close the program with <span style="color:#228B22">'''Q'''</span> or <span style="color:#228B22">'''Alt+F4'''</span>.

Next, run it again giving the following parameter:

<syntaxhighlight lang=Bash>./openniViewer -v inputCloud0.pcd</syntaxhighlight>

This will tell the program not to take data from the device, but from the saved point cloud file instead. After it loads, you will realize that you are presented the same scene you saved to disk.

<span style="color:#606060">'''''NOTE: PCD data is saved relative to the sensor. No matter how much you have manipulated the view, it will reset to default when you load the file.'''''</span>

=Conclusion=

At this point, your Kinect device should be working and getting depth data for you. There is a collection of excellent [http://pointclouds.org/documentation/tutorials/ tutorials] for PCL in the official webpage. I encourage you to finish them all before proceeding your experiments with the Kinect sensor. You can also find a good introduction/tutorial at the PCL library [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Introduction.pdf here].

If you use an ASUS Xtion PRO device instead, you should have gotten everything to work without problems or additional steps (except maybe for [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO | this one]]).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 1: Installing and testing

2015-11-01T22:57:28Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Kinect.jpg|thumb|right|200px|Microsoft Kinect device.]]

This series of tutorials will explain the usage of a depth camera like [https://en.wikipedia.org/wiki/Kinect Kinect] for "serious" researching purposes. As you may know, Kinect is in fact an affordable depth sensor, developed with technology from [https://en.wikipedia.org/wiki/PrimeSense PrimeSense], based on infrarred structured light method. It also has a common camera (which makes it a RGB-D device), a microphone and a motorized pivot. Its use is not limited to playing with a Xbox360 console, you can plug it to a computer and use it like any other sensor. Many open-source drivers and frameworks are available.

Since its release on November 2010, it has gained a lot of popularity, specially among the scientific community. Many researches have procured themselves one because, despite the low cost (about 100 $), it has proven to be a powerful solution for depth sensing projects. Current investigations focus on real-time surface mapping, object recognition and tracking, and localization. Impressive results (like the [http://research.microsoft.com/en-us/projects/surfacerecon/ KinectFusion] project from Microsoft) are already possible.

The new [https://en.wikipedia.org/wiki/Xbox_One Xbox One] ships with an upgraded version, [https://en.wikipedia.org/wiki/Kinect_for_Xbox_One Kinect v2], with enhanced resolution, that is able to detect your facial expression, measure your heart rate, and track every one of your fingers. A PC development-ready version ([https://en.wikipedia.org/wiki/Kinect_for_Xbox_One#Kinect_for_Windows Kinect for Windows v2]) was released in July 2014, but it could only be used with the official Windows SDK (open source support [https://github.com/OpenKinect/libfreenect2 exists] but is still young). In October 2014 an adapter that allows to plug the standard Kinect v2 to a PC was released, so the development version of the sensor was discontinued in April 2015. Now you can just buy the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-for-Xbox-One/productID.307499400 standard one] for the console plus the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600 adapter].

I will explain the installation and usage of one of the "legacy" Kinect 1.0 devices with a common PC, and the possibilities it offers. I will do it in an easy to understand way, intended for students that have just acquired it and want to start from scratch.

Keep in mind that the software that we are going to use (Point Cloud Library and OpenNI drivers) will also let you use any other device like the [https://www.asus.com/3D-Sensor/Xtion_PRO/ Xtion PRO] or [https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/ Xtion PRO LIVE] from ASUS (the PRO only has a depth sensor, the PRO LIVE is a RGB-D camera) without changing a line of code.

<span style="color:#606060">'''''NOTE: The tutorials are written for Linux platforms. Also, 64-bit versions seem to work better than 32-bit.'''''</span>

=Requirements=

You will need the following:

* A common Kinect device, out of the box. You can buy it in your local electronics shop, or online. It also includes a free copy of ''[https://en.wikipedia.org/wiki/Kinect_Adventures! Kinect Adventures!]'', which is useless if you do not own the console. Microsoft has released a ''Kinect for Windows'' device, which is a normal looking Kinect no longer compatible with Xbox360, that will only work with their official SDK, intended for developers only. Also, like I stated earlier, you can use an ASUS Xtion indistinctly.
* A computer running Linux (Debian or Ubuntu preferably).
* A medium-sized room. Kinect has some limitations for depth measurement: 40cm minimum, 8m maximum (make it 6).

<span style="color:#606060">'''''NOTE: ''Kinect for Windows'' may have problems working with open source drivers on Linux .'''''</span>

=Connecting everything=

Kinect does not work with a common USB port. Its power consumption is a bit higher because of the motor, so Microsoft came up with a connector that combines USB and power supply. Old Xbox 360 models needed a special adapter, new S model ones already have this new port. Luckily, Kinect comes with the official adapter out of the box (otherwise you will have to [http://www.amazon.com/Power-Supply-Cable-Kinect-Xbox-360/dp/B004S7GA46/ref=sr_1_1?ie=UTF8&qid=1378288327&sr=8-1 buy one]).

Just plug the adapter to any power socket, and the USB to your computer. Let's check typing this in a terminal:

<syntaxhighlight lang=Bash>lsusb</syntaxhighlight>

Output should list the following devices:

<syntaxhighlight lang=Bash>Bus 001 Device 005: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 006: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 007: ID 045e:02ae Microsoft Corp. Xbox NUI Camera</syntaxhighlight>

If you are using a Xtion, you should see:

<syntaxhighlight lang=Bash>Bus 001 Device 004: ID 1d27:0601 ASUS</syntaxhighlight>

"0601" identifies the new Xtion model, "0600" the older one. Both should work the same. But try to avoid [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO|USB 3.0 ports]]!

=Installing the software=

There is more than one way to get your Kinect working on your PC, and start developing applications for it:

* [https://www.microsoft.com/en-us/download/details.aspx?id=40278 Kinect for Windows SDK] and [https://www.microsoft.com/en-us/download/details.aspx?id=40276 Developer Toolkit]: released on June 16, 2011 as a non-commercial SDK intended for application development. Version 1.8, the last there will ever be now that Kinect v2 is out, was released on September 2013. Because it comes from Microsoft, it is obviously the easiest way to get everything working. Sadly, there is no Linux version.
* [https://github.com/OpenKinect/libfreenect libfreenect] library: from the [http://openkinect.org/wiki/Main_Page OpenKinect] project, it is intended to be a free and open source alternative to the official drivers. libfreenect is used by projects like [https://github.com/ofTheo/ofxKinect ofxKinect], an addon for the [http://www.openframeworks.cc/ openFrameworks] toolkit (and as of version 0.8, included in the core package) that runs on Linux and OS X. ofxKinect packs a nice example application to show the RGB and point cloud taken from Kinect.
* [https://github.com/PrimeSense/Sensor PrimeSense drivers]: they were released as open source after the the [https://en.wikipedia.org/wiki/OpenNI OpenNI] project was created, along with the motion tracking middleware, ''NITE'', and the SDK. NI stands for Natural Interaction, and the project tried to enforce a common standard for human input using Kinect-like sensors. These official drivers are used by [http://wiki.ros.org/openni_kinect ROS] (the Robot Operating System, a massive collection of libraries and tools for robotic researchers) and [http://pointclouds.org/ PCL] (the Point Cloud Library, with everything needed for 3D point cloud processing). Sadly, version 2.0 of the OpenNI SDK dropped support for Kinect on Linux due to licensing issues, so for now PCL relies on legacy 1.x versions. Also, Apple bought PrimeSense on November 2013, and on April 2014 OpenNI's webpage was closed. The source is now being maintained by [http://structure.io/openni a third party].
* [https://github.com/avin2/SensorKinect SensorKinect]: a modified version of the official PrimeSense drivers, used for example by [https://github.com/gameoverhack/ofxOpenNI ofxOpenNI] (another openFrameworks addon). Last updated on 2012.

For this tutorial, we are going to use PCL with the OpenNI drivers, so owners of a Xtion can also get it to work.

==Precompiled PCL for Ubuntu==

If you have a valid [http://wiki.ros.org/ROS/Installation installation of ROS] (through their repository), you do not have to install anything. Both OpenNI and PrimeSense drivers, as well as PCL, should be already installed. You can check it with:

<syntaxhighlight lang=Bash>sudo apt-get install libpcl-1.7-all libpcl-1.7-all-dev libopenni-dev libopenni-sensor-primesense-dev</syntaxhighlight>

The previous command should state that all packages are already installed (change the PCL version number as needed), install them if not. If you get an error about overwriting some file, check [[PCL/OpenNI_troubleshooting#Trying_to_overwrite_X.2C_which_is_also_in_package_Y|this]].

In case you do not want ROS, there is a [https://launchpad.net/~v-launchpad-jochen-sprickerhof-de/+archive/ubuntu/pcl PPA] (Personal Package Archive, a private repository) which has everything we need. Add it to your sources, and install everything:

<syntaxhighlight lang=Bash>sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install build-essential cmake libpcl1.7 libpcl-dev pcl-tools</syntaxhighlight>

Trying to mix ROS and PCL repositories and packages can cause some errors, so choose one of them and stick with it. Check the [[PCL/OpenNI troubleshooting]] page because your Kinect may not work by default in 32 bits.

==Compiling PCL from source==

For Linuxes without a precompiled version of PCL, you will need to compile it yourself. This has an advantage, actually: you can customize the build options and choose what you want. Also, the resulting binaries and libraries should be a bit faster. And you always get the latest version! The instructions are [http://pointclouds.org/documentation/tutorials/compiling_pcl_posix.php here], but the steps are easy so I will show them to you.

===Installing the dependencies===

Some of PCL dependencies can be installed via the package manager. Others will require additional work.

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install build-essential cmake cmake-curses-gui libboost-all-dev libeigen3-dev libflann-dev libvtk5-dev libvtk5-qt4-dev libglew-dev libxmu-dev libsuitesparse-dev libqhull-dev libpcap-dev libxmu-dev libxi-dev libgtest-dev libqt4-opengl-dev</syntaxhighlight>

The trunk version of PCL uses [https://en.wikipedia.org/wiki/VTK VTK] 6 and [https://en.wikipedia.org/wiki/Qt_%28software%29 Qt] 5, so if you intend to compile it, you must install the following packages (say yes if you are asked to remove VTK 5 and Qt 4 packages):

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

====JDK====

OpenNI requires Sun's official JDK (Java Development Kit), which is no longer available on official apt repositories. You can use [http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html unofficial ones], or do it manually. For the last method, go to the [http://www.oracle.com/technetwork/java/javase/downloads/index.html Java SE downloads] page (SE means Standard Edition) and download the latest version (e.g., <span style="color:#1E90FF">''jdk-8u66-linux-x64.tar.gz''</span>). Extract it, then move the contents to <span style="color:#FF8C00">''/usr/lib/jvm/''</span> so it is available system-wide:

<syntaxhighlight lang=Bash>sudo mkdir -p /usr/lib/jvm/
sudo cp -r jdk1.8.0_66/ /usr/lib/jvm/</syntaxhighlight>

Then, make it the default choice to compile and run Java programs. Remember to change the version number as needed!

<syntaxhighlight lang=Bash>sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_66/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_66/bin/javac" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.8.0_66/bin/jar" 1</syntaxhighlight>

To be sure, use the following commands to manually select the correct option, in case there is more than one choice:

<syntaxhighlight lang=Bash>sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config jar</syntaxhighlight>

If you are still not sure, run them and display the version, making sure it is the one you installed:

<syntaxhighlight lang=Bash>java -version
javac -version</syntaxhighlight>

Sun's JDK is now installed.

====OpenNI====

PCL uses OpenNI and the PrimeSense drivers to get data from devices like Kinect or Xtion. It is optional, but it would not make much sense not to install it, would it? If you are using Ubuntu, add the [[#Precompiled PCL for Ubuntu | PPA]] and install <span style="color:#228B22">''libopenni-dev''</span> and <span style="color:#228B22">''libopenni-sensor-primesense-dev''</span>, which should be done already. Otherwise, fetch the [https://github.com/OpenNI/OpenNI OpenNI] and [https://github.com/PrimeSense/Sensor PrimeSense Sensor] sources from GitHub (download them as .zip, the link is on the right). Extract them, and install the dependencies:

<syntaxhighlight lang=Bash>sudo apt-get install python libusb-1.0-0-dev freeglut3-dev doxygen graphviz</syntaxhighlight>

Go to the directory where you extracted OpenNI (<span style="color:#FF8C00">''OpenNI-master/''</span> for me), and open a terminal in the <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span> subdirectory. Issue:

<syntaxhighlight lang=Bash>./RedistMaker</syntaxhighlight>

When it finishes, and if there are no errors (check the [[PCL/OpenNI troubleshooting]] page if you get any), go to <span style="color:#FF8C00">''Platform/Linux/Redist/OpenNI-Bin-Dev-Linux-x64-v1.5.7.10/''</span> (or your equivalent), and install (you must be root):

<syntaxhighlight lang=Bash>sudo ./install.sh</syntaxhighlight>

Now, go to the directory where you extracted the PrimeSense drivers (<span style="color:#FF8C00">''Sensor-master/''</span> for me), and repeat the exact same procedure (go to <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span>, issue <span style="color:#228B22">''./RedistMaker''</span>, go to <span style="color:#FF8C00">''Platform/Linux/Redist/Sensor-Bin-Linux-x64-v5.1.6.6/''</span>, issue <span style="color:#228B22">''sudo ./install.sh''</span>). Congratulations, you have now installed OpenNI.

====CUDA====

Like OpenNI, [https://en.wikipedia.org/wiki/CUDA nVidia CUDA] is an optional dependency, that will allow PCL to use your GPU (Graphics Processing Unit, that is, your graphics card) for certain computations. This is mandatory for tools like KinFu (do not bother unless you have at least a series 400 card with 1.5 GB of VRAM).

Some distributions provide packages for CUDA in the repositories. For example, in Ubuntu:

<syntaxhighlight lang=Bash>sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit</syntaxhighlight>

If you want to install it manually (which is incompatible with the previous method), go to the [https://developer.nvidia.com/cuda-downloads CUDA downloads] page, which is self-explanatory, and get the small .deb file or the huge .run toolkit/SDK installer file for your system (you should have installed the nVidia drivers already, but the installer will also do it for you if needed).

If you chose the .deb file, install it using the method of your choice, like Gdebi package manager, or through console:

<syntaxhighlight lang=Bash>sudo dpkg -i <package.deb></syntaxhighlight>

The .deb does not contain all CUDA stuff, it just adds their repository to your software lists. Now you must install everything:

<syntaxhighlight lang=Bash>sudo apt-get update
sudo apt-get install cuda</syntaxhighlight>

If, on the other hand, you downloaded the .run, give it execution permissions:

<syntaxhighlight lang=Bash>chmod +x cuda_7.0.28_linux.run</syntaxhighlight>

And install it. You can use the default options, although if you have a working nVidia graphics driver for your card, you may want to say "no" when the installer offers to install it for you:

<syntaxhighlight lang=Bash>sudo ./cuda_7.0.28_linux.run</syntaxhighlight>

The environment variables need to be changed so your system can find CUDA's libraries and binaries. Open <span style="color:#1E90FF">''/etc/ld.so.conf''</span>:

<syntaxhighlight lang=Bash>sudo nano /etc/ld.so.conf</syntaxhighlight>

And append one of these two lines:

<syntaxhighlight lang=Bash>/usr/local/cuda/lib64 # Add this on 64-bit only.
/usr/local/cuda/lib # Add this on 32-bit only.</syntaxhighlight>

Save with <span style="color:#228B22">'''Ctrl+O'''</span> and Enter, exit with <span style="color:#228B22">'''Ctrl+X'''</span>. Reload the cache of the dynamic linker with:

<syntaxhighlight lang=Bash>sudo ldconfig</syntaxhighlight>

Now, append CUDA's bin directory to your PATH. Do this by editing your local <span style="color:#1E90FF">''.bashrc''</span> file:

<syntaxhighlight lang=Bash>nano ~/.bashrc</syntaxhighlight>

And append this line:

<syntaxhighlight lang=Bash>export PATH=$PATH:/usr/local/cuda/bin</syntaxhighlight>

CUDA is now installed.

===Getting the source===

Every dependency is installed. Time to download PCL's source code. First, you must choose whether to install the stable or the experimental branch of PCL. The stable branch is the latest official release and it is guaranteed to work without problems. The experimental branch may give you a compiling error seldomly, but you can find some interesting features that stable users will have to wait some months for. Apart from that, both are built the same way.

To get the stable version, go to the [https://github.com/PointCloudLibrary/pcl/releases downloads] page, get <span style="color:#1E90FF">''pcl-pcl-1.7.2.tar.gz''</span> or whatever the latest release is, and extract it somewhere. For the experimental version, use Git:

<syntaxhighlight lang=Bash>sudo apt-get install git
git clone https://github.com/PointCloudLibrary/pcl PCL-trunk-Source</syntaxhighlight>

The compiled trunk version of PCL will take up more than 8 GB. So make sure you put the source folder in a partition with enough free space!

===Compiling===

Go the the PCL source directory (<span style="color:#FF8C00">''pcl-pcl-1.7.2/''</span> or <span style="color:#FF8C00">''PCL-trunk-Source/''</span>), and create a new subdirectory to keep the build files in:

<syntaxhighlight lang=Bash>mkdir build
cd build</syntaxhighlight>

Now it is time to configure the project using CMake. We will tell it to build in Release (fully optimized, no debug capabilities) mode now, and customize the rest of the options later:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..</syntaxhighlight>

CMake should be able to find every dependency, thus being able to build every subsystem except for the ones marked as <span style="color:#FF1493">''"Disabled by default"''</span>. If you are happy, you can build now, otherwise let's invoke CMake's curses interface to change a couple of things (mind the final dot):

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

[[Image:Ccmake_GUI_PCL.png|thumb|center|900px|Interface of ''ccmake''.]]

Here you can change the build options. The program usage can be found at the bottom of the screen. Try turning all functionality <span style="color:#FF1493">"ON"</span>. The most important thing, in case you want to use CUDA, is to enable it and give CMake the path to your SDK. Go to the <span style="color:#FF1493">"CUDA_SDK_ROOT_DIR"</span> option and enter the correct path (probably <span style="color:#FF8C00">''/usr/local/cuda/''</span> or similar).

When you are done, press <span style="color:#228B22">'''C'''</span> to configure and <span style="color:#228B22">'''G'''</span> to generate and exit the tool. Sometimes, the options you change can activate previously omitted parameters, or prompt some warning text. Just press <span style="color:#228B22">'''E'''</span> when you are finished reading the message, and keep pressing <span style="color:#228B22">'''C'''</span> until it lets you generate (new parameters will be marked with an asterisk, so you can check them and decide whether or not you want further customization).

If you are done configuring, it is time to build:

<syntaxhighlight lang=Bash>make</syntaxhighlight>

<span style="color:#606060">'''''NOTE: Additionally, you can append the parameter -jX to speed up the compilation, X being the number of cores or processors of your PC, plus one.'''''</span>

Remember that, at any time, you can manually force the project to be reconfigured and built from scratch by emptying the <span style="color:#FF8C00">''build/''</span> directory with:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

===Installing===

It will take some time to compile PCL (up to a few hours if your PC is not powerful enough). When it is finished, install it system-wide with:

<syntaxhighlight lang=Bash>sudo make install</syntaxhighlight>

And you should reboot and proceed to the next section, to see if your computer now recognizes (and uses) your Kinect device.

=Testing (OpenNI viewer)=

We are going to write a simple example program that will fetch data from the Kinect or Xtion and present it to the user, using the PCL library. It will also allow to save the current frame (as point cloud) to disk. If you feel lazy, you can download it [http://robotica.unileon.es/~victorm/PCL_openni_viewer.tar.gz here], and skip the next two sections. Otherwise, create a new directory anywhere in your hard disk and proceed.

==CMakeLists.txt==

Inside that directory, create a new text file named <span style="color:#1E90FF">''CMakeLists.txt''</span>. PCL-based programs use the CMake build system, too. Open it with any editor and paste the following content:

<syntaxhighlight lang=CMake>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

project(PCL_openni_viewer)

find_package(PCL 1.7 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

set(PCL_BUILD_TYPE Release)

file(GLOB PCL_openni_viewer_SRC
"src/*.h"
"src/*.cpp"
)
add_executable(openniViewer ${PCL_openni_viewer_SRC})

target_link_libraries (openniViewer ${PCL_LIBRARIES})</syntaxhighlight>

CMake syntax is quite self-explanatory. We ask for a CMake version 2.8 installation, minimum. We declare a new project named <span style="color:#FF1493">"openni_PCL_viewer"</span>. We tell CMake to check for the presence of PCL library development files, version 1.7. If our system can not meet the CMake and PCL version requirement, the process will fail.

Next, we feed the compiler and linker the directories where PCL includes and libraries can be found, and the defined symbols. We tell CMake to use the <span style="color:#FF1493">"Release"</span> build type, which will activate certain optimizations depending on the compiler we use. Other build types are available, like <span style="color:#FF1493">"Debug"</span>, <span style="color:#FF1493">"MinSizeRel"</span>, and <span style="color:#FF1493">"RelWithDebInfo"</span>.

Finally, we create a variable, <span style="color:#FF1493">"opennipclviewer_SRC"</span>, that will store a list of files to be compiled (though we will only have one). We create a new binary to be compiled from these source files, and we link it with the PCL library.

Check the [http://www.cmake.org/Wiki/CMake_Useful_Variables CMake help] for more interesting options.

==main.cpp==

We told CMake it could find the source files in a <span style="color:#FF8C00">''src/''</span> subdirectory, so let's keep to out word and create it. Then, add a new <span style="color:#1E90FF">''main.cpp''</span> file inside and paste the following lines:

<syntaxhighlight lang=CPP>// Original code by Geoffrey Biggs, taken from the PCL tutorial in
// http://pointclouds.org/documentation/tutorials/pcl_visualizer.php

// Simple OpenNI viewer that also allows to write the current scene to a .pcd
// when pressing SPACE.

#include <pcl/io/openni_grabber.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/console/parse.h>

#include <iostream>

using namespace std;
using namespace pcl;

PointCloud<PointXYZRGBA>::Ptr cloudptr(new PointCloud<PointXYZRGBA>); // A cloud that will store color info.
PointCloud<PointXYZ>::Ptr fallbackCloud(new PointCloud<PointXYZ>); // A fallback cloud with just depth data.
boost::shared_ptr<visualization::CloudViewer> viewer; // Point cloud viewer object.
Grabber* openniGrabber; // OpenNI grabber that takes data from the device.
unsigned int filesSaved = 0; // For the numbering of the clouds saved to disk.
bool saveCloud(false), noColor(false); // Program control.

void
printUsage(const char* programName)
{
cout << "Usage: " << programName << " [options]"
<< endl
<< endl
<< "Options:\n"
<< endl
<< "\t<none> start capturing from an OpenNI device.\n"
<< "\t-v FILE visualize the given .pcd file.\n"
<< "\t-h shows this help.\n";
}

// This function is called every time the device has new data.
void
grabberCallback(const PointCloud<PointXYZRGBA>::ConstPtr& cloud)
{
if (! viewer->wasStopped())
viewer->showCloud(cloud);

if (saveCloud)
{
stringstream stream;
stream << "inputCloud" << filesSaved << ".pcd";
string filename = stream.str();
if (io::savePCDFile(filename, *cloud, true) == 0)
{
filesSaved++;
cout << "Saved " << filename << "." << endl;
}
else PCL_ERROR("Problem saving %s.\n", filename.c_str());

saveCloud = false;
}
}

// For detecting when SPACE is pressed.
void
keyboardEventOccurred(const visualization::KeyboardEvent& event,
void* nothing)
{
if (event.getKeySym() == "space" && event.keyDown())
saveCloud = true;
}

// Creates, initializes and returns a new viewer.
boost::shared_ptr<visualization::CloudViewer>
createViewer()
{
boost::shared_ptr<visualization::CloudViewer> v
(new visualization::CloudViewer("OpenNI viewer"));
v->registerKeyboardCallback(keyboardEventOccurred);

return (v);
}

int
main(int argc, char** argv)
{
if (console::find_argument(argc, argv, "-h") >= 0)
{
printUsage(argv[0]);
return -1;
}

bool justVisualize(false);
string filename;
if (console::find_argument(argc, argv, "-v") >= 0)
{
if (argc != 3)
{
printUsage(argv[0]);
return -1;
}

filename = argv[2];
justVisualize = true;
}
else if (argc != 1)
{
printUsage(argv[0]);
return -1;
}

// First mode, open and show a cloud from disk.
if (justVisualize)
{
// Try with color information...
try
{
io::loadPCDFile<PointXYZRGBA>(filename.c_str(), *cloudptr);
}
catch (PCLException e1)
{
try
{
// ...and if it fails, fall back to just depth.
io::loadPCDFile<PointXYZ>(filename.c_str(), *fallbackCloud);
}
catch (PCLException e2)
{
return -1;
}

noColor = true;
}

cout << "Loaded " << filename << "." << endl;
if (noColor)
cout << "This cloud has no RGBA color information present." << endl;
else cout << "This cloud has RGBA color information present." << endl;
}
// Second mode, start fetching and displaying frames from the device.
else
{
openniGrabber = new OpenNIGrabber();
if (openniGrabber == 0)
return -1;
boost::function<void (const PointCloud<PointXYZRGBA>::ConstPtr&)> f =
boost::bind(&grabberCallback, _1);
openniGrabber->registerCallback(f);
}

viewer = createViewer();

if (justVisualize)
{
if (noColor)
viewer->showCloud(fallbackCloud);
else viewer->showCloud(cloudptr);
}
else openniGrabber->start();

// Main loop.
while (! viewer->wasStopped())
boost::this_thread::sleep(boost::posix_time::seconds(1));

if (! justVisualize)
openniGrabber->stop();
}</syntaxhighlight>

Save and close.

==Compiling==

Follow the same steps you used to build PCL. That is, create a new <span style="color:#FF8C00">''build/''</span> subdirectory next to the <span style="color:#FF8C00">''src/''</span> one. Open a terminal there and issue:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..
make</syntaxhighlight>

==Executing==

Still from the same terminal, run the compiled example program:

<syntaxhighlight lang=Bash>./openniViewer</syntaxhighlight>

After some seconds, the main window will appear and the application will start grabbing frames from the device. You can inspect the current point cloud using the mouse, holding the left button to rotate, the right one (or the mouse wheel) to zoom, and the middle one to pan the camera around. At first, you may see only a black screen, or some big colored axes, but no cloud. Try zooming out, to see the whole scene. Another useful key is <span style="color:#228B22">'''R'''</span>, which will reset the camera parameters when pressed. Use it whenever you notice that zooming has gotten slow after some camera movement, or if you still can not see the cloud. See the [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer] tutorial for additional controls and features.

Whenever you feel ready, press the <span style="color:#228B22">'''SPACE'''</span> key. The program will pause for a fraction of a second and the output <span style="color:#FF1493">''"Saved inputCloud0.pcd."''</span> will appear on the console. Check the current folder to see that file <span style="color:#1E90FF">''inputCloud0.pcd''</span> has indeed been written. You can now close the program with <span style="color:#228B22">'''Q'''</span> or <span style="color:#228B22">'''Alt+F4'''</span>.

Next, run it again giving the following parameter:

<syntaxhighlight lang=Bash>./openniViewer -v inputCloud0.pcd</syntaxhighlight>

This will tell the program not to take data from the device, but from the saved point cloud file instead. After it loads, you will realize that you are presented the same scene you saved to disk.

<span style="color:#606060">'''''NOTE: PCD data is saved relative to the sensor. No matter how much you have manipulated the view, it will reset to default when you load the file.'''''</span>

=Conclusion=

At this point, your Kinect device should be working and getting depth data for you. There is a collection of excellent [http://pointclouds.org/documentation/tutorials/ tutorials] for PCL in the official webpage. I encourage you to finish them all before proceeding your experiments with the Kinect sensor. You can also find a good introduction/tutorial at the PCL library [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Introduction.pdf here].

If you use an ASUS Xtion PRO device instead, you should have gotten everything to work without problems or additional steps (except maybe for [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO | this one]]).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 1: Installing and testing

2015-11-01T22:55:47Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Kinect.jpg|thumb|right|200px|Microsoft Kinect device.]]

This series of tutorials will explain the usage of a depth camera like [https://en.wikipedia.org/wiki/Kinect Kinect] for "serious" researching purposes. As you may know, Kinect is in fact an affordable depth sensor, developed with technology from [https://en.wikipedia.org/wiki/PrimeSense PrimeSense], based on infrarred structured light method. It also has a common camera (which makes it a RGB-D device), a microphone and a motorized pivot. Its use is not limited to playing with a Xbox360 console, you can plug it to a computer and use it like any other sensor. Many open-source drivers and frameworks are available.

Since its release on November 2010, it has gained a lot of popularity, specially among the scientific community. Many researches have procured themselves one because, despite the low cost (about 100 $), it has proven to be a powerful solution for depth sensing projects. Current investigations focus on real-time surface mapping, object recognition and tracking, and localization. Impressive results (like the [http://research.microsoft.com/en-us/projects/surfacerecon/ KinectFusion] project from Microsoft) are already possible.

The new [https://en.wikipedia.org/wiki/Xbox_One Xbox One] ships with an upgraded version, [https://en.wikipedia.org/wiki/Kinect_for_Xbox_One Kinect v2], with enhanced resolution, that is able to detect your facial expression, measure your heart rate, and track every one of your fingers. A PC development-ready version ([https://en.wikipedia.org/wiki/Kinect_for_Xbox_One#Kinect_for_Windows Kinect for Windows v2]) was released in July 2014, but it could only be used with the official Windows SDK (open source support [https://github.com/OpenKinect/libfreenect2 exists] but is still young). In October 2014 an adapter that allows to plug the standard Kinect v2 to a PC was released, so the development version of the sensor was discontinued in April 2015. Now you can just buy the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-for-Xbox-One/productID.307499400 standard one] for the console plus the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600 adapter].

I will explain the installation and usage of one of the "legacy" Kinect 1.0 devices with a common PC, and the possibilities it offers. I will do it in an easy to understand way, intended for students that have just acquired it and want to start from scratch.

Keep in mind that the software that we are going to use (Point Cloud Library and OpenNI drivers) will also let you use any other device like the [https://www.asus.com/3D-Sensor/Xtion_PRO/ Xtion PRO] or [https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/ Xtion PRO LIVE] from ASUS (the PRO only has a depth sensor, the PRO LIVE is a RGB-D camera) without changing a line of code.

<span style="color:#606060">'''''NOTE: The tutorials are written for Linux platforms. Also, 64-bit versions seem to work better than 32-bit.'''''</span>

=Requirements=

You will need the following:

* A common Kinect device, out of the box. You can buy it in your local electronics shop, or online. It also includes a free copy of ''[https://en.wikipedia.org/wiki/Kinect_Adventures! Kinect Adventures!]'', which is useless if you do not own the console. Microsoft has released a ''Kinect for Windows'' device, which is a normal looking Kinect no longer compatible with Xbox360, that will only work with their official SDK, intended for developers only. Also, like I stated earlier, you can use an ASUS Xtion indistinctly.
* A computer running Linux (Debian or Ubuntu preferably).
* A medium-sized room. Kinect has some limitations for depth measurement: 40cm minimum, 8m maximum (make it 6).

<span style="color:#606060">'''''NOTE: ''Kinect for Windows'' may have problems working with open source drivers on Linux .'''''</span>

=Connecting everything=

Kinect does not work with a common USB port. Its power consumption is a bit higher because of the motor, so Microsoft came up with a connector that combines USB and power supply. Old Xbox 360 models needed a special adapter, new S model ones already have this new port. Luckily, Kinect comes with the official adapter out of the box (otherwise you will have to [http://www.amazon.com/Power-Supply-Cable-Kinect-Xbox-360/dp/B004S7GA46/ref=sr_1_1?ie=UTF8&qid=1378288327&sr=8-1 buy one]).

Just plug the adapter to any power socket, and the USB to your computer. Let's check typing this in a terminal:

<syntaxhighlight lang=Bash>lsusb</syntaxhighlight>

Output should list the following devices:

<syntaxhighlight lang=Bash>Bus 001 Device 005: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 006: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 007: ID 045e:02ae Microsoft Corp. Xbox NUI Camera</syntaxhighlight>

If you are using a Xtion, you should see:

<syntaxhighlight lang=Bash>Bus 001 Device 004: ID 1d27:0601 ASUS</syntaxhighlight>

"0601" identifies the new Xtion model, "0600" the older one. Both should work the same. But try to avoid [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO|USB 3.0 ports]]!

=Installing the software=

There is more than one way to get your Kinect working on your PC, and start developing applications for it:

* [https://www.microsoft.com/en-us/download/details.aspx?id=40278 Kinect for Windows SDK] and [https://www.microsoft.com/en-us/download/details.aspx?id=40276 Developer Toolkit]: released on June 16, 2011 as a non-commercial SDK intended for application development. Version 1.8, the last there will ever be now that Kinect v2 is out, was released on September 2013. Because it comes from Microsoft, it is obviously the easiest way to get everything working. Sadly, there is no Linux version.
* [https://github.com/OpenKinect/libfreenect libfreenect] library: from the [http://openkinect.org/wiki/Main_Page OpenKinect] project, it is intended to be a free and open source alternative to the official drivers. libfreenect is used by projects like [https://github.com/ofTheo/ofxKinect ofxKinect], an addon for the [http://www.openframeworks.cc/ openFrameworks] toolkit (and as of version 0.8, included in the core package) that runs on Linux and OS X. ofxKinect packs a nice example application to show the RGB and point cloud taken from Kinect.
* [https://github.com/PrimeSense/Sensor PrimeSense drivers]: they were released as open source after the the [https://en.wikipedia.org/wiki/OpenNI OpenNI] project was created, along with the motion tracking middleware, ''NITE'', and the SDK. NI stands for Natural Interaction, and the project tried to enforce a common standard for human input using Kinect-like sensors. These official drivers are used by [http://wiki.ros.org/openni_kinect ROS] (the Robot Operating System, a massive collection of libraries and tools for robotic researchers) and [http://pointclouds.org/ PCL] (the Point Cloud Library, with everything needed for 3D point cloud processing). Sadly, version 2.0 of the OpenNI SDK dropped support for Kinect on Linux due to licensing issues, so for now PCL relies on legacy 1.x versions. Also, Apple bought PrimeSense on November 2013, and on April 2014 OpenNI's webpage was closed. The source is now being maintained by [http://structure.io/openni a third party].
* [https://github.com/avin2/SensorKinect SensorKinect]: a modified version of the official PrimeSense drivers, used for example by [https://github.com/gameoverhack/ofxOpenNI ofxOpenNI] (another openFrameworks addon). Last updated on 2012.

For this tutorial, we are going to use PCL with the OpenNI drivers, so owners of a Xtion can also get it to work.

==Precompiled PCL for Ubuntu==

If you have a valid installation of [http://wiki.ros.org/indigo/Installation/Ubuntu ROS] (through their repository), you do not have to install anything. Both OpenNI and PrimeSense drivers, as well as PCL, should be already installed. You can check it with:

<syntaxhighlight lang=Bash>sudo apt-get install libpcl-1.7-all libpcl-1.7-all-dev libopenni-dev libopenni-sensor-primesense-dev</syntaxhighlight>

The previous command should state that all packages are already installed (change the PCL version number as needed), install them if not. If you get an error about overwriting some file, check [[PCL/OpenNI_troubleshooting#Trying_to_overwrite_X.2C_which_is_also_in_package_Y|this]].

In case you do not want ROS, there is a [https://launchpad.net/~v-launchpad-jochen-sprickerhof-de/+archive/ubuntu/pcl PPA] (Personal Package Archive, a private repository) which has everything we need. Add it to your sources, and install everything:

<syntaxhighlight lang=Bash>sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install build-essential cmake libpcl1.7 libpcl-dev pcl-tools</syntaxhighlight>

Trying to mix ROS and PCL repositories and packages can cause some errors, so choose one of them and stick with it. Check the [[PCL/OpenNI troubleshooting]] page because your Kinect may not work by default in 32 bits.

==Compiling PCL from source==

For Linuxes without a precompiled version of PCL, you will need to compile it yourself. This has an advantage, actually: you can customize the build options and choose what you want. Also, the resulting binaries and libraries should be a bit faster. And you always get the latest version! The instructions are [http://pointclouds.org/documentation/tutorials/compiling_pcl_posix.php here], but the steps are easy so I will show them to you.

===Installing the dependencies===

Some of PCL dependencies can be installed via the package manager. Others will require additional work.

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install build-essential cmake cmake-curses-gui libboost-all-dev libeigen3-dev libflann-dev libvtk5-dev libvtk5-qt4-dev libglew-dev libxmu-dev libsuitesparse-dev libqhull-dev libpcap-dev libxmu-dev libxi-dev libgtest-dev libqt4-opengl-dev</syntaxhighlight>

The trunk version of PCL uses [https://en.wikipedia.org/wiki/VTK VTK] 6 and [https://en.wikipedia.org/wiki/Qt_%28software%29 Qt] 5, so if you intend to compile it, you must install the following packages (say yes if you are asked to remove VTK 5 and Qt 4 packages):

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

====JDK====

OpenNI requires Sun's official JDK (Java Development Kit), which is no longer available on official apt repositories. You can use [http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html unofficial ones], or do it manually. For the last method, go to the [http://www.oracle.com/technetwork/java/javase/downloads/index.html Java SE downloads] page (SE means Standard Edition) and download the latest version (e.g., <span style="color:#1E90FF">''jdk-8u66-linux-x64.tar.gz''</span>). Extract it, then move the contents to <span style="color:#FF8C00">''/usr/lib/jvm/''</span> so it is available system-wide:

<syntaxhighlight lang=Bash>sudo mkdir -p /usr/lib/jvm/
sudo cp -r jdk1.8.0_66/ /usr/lib/jvm/</syntaxhighlight>

Then, make it the default choice to compile and run Java programs. Remember to change the version number as needed!

<syntaxhighlight lang=Bash>sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_66/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_66/bin/javac" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.8.0_66/bin/jar" 1</syntaxhighlight>

To be sure, use the following commands to manually select the correct option, in case there is more than one choice:

<syntaxhighlight lang=Bash>sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config jar</syntaxhighlight>

If you are still not sure, run them and display the version, making sure it is the one you installed:

<syntaxhighlight lang=Bash>java -version
javac -version</syntaxhighlight>

Sun's JDK is now installed.

====OpenNI====

PCL uses OpenNI and the PrimeSense drivers to get data from devices like Kinect or Xtion. It is optional, but it would not make much sense not to install it, would it? If you are using Ubuntu, add the [[#Precompiled PCL for Ubuntu | PPA]] and install <span style="color:#228B22">''libopenni-dev''</span> and <span style="color:#228B22">''libopenni-sensor-primesense-dev''</span>, which should be done already. Otherwise, fetch the [https://github.com/OpenNI/OpenNI OpenNI] and [https://github.com/PrimeSense/Sensor PrimeSense Sensor] sources from GitHub (download them as .zip, the link is on the right). Extract them, and install the dependencies:

<syntaxhighlight lang=Bash>sudo apt-get install python libusb-1.0-0-dev freeglut3-dev doxygen graphviz</syntaxhighlight>

Go to the directory where you extracted OpenNI (<span style="color:#FF8C00">''OpenNI-master/''</span> for me), and open a terminal in the <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span> subdirectory. Issue:

<syntaxhighlight lang=Bash>./RedistMaker</syntaxhighlight>

When it finishes, and if there are no errors (check the [[PCL/OpenNI troubleshooting]] page if you get any), go to <span style="color:#FF8C00">''Platform/Linux/Redist/OpenNI-Bin-Dev-Linux-x64-v1.5.7.10/''</span> (or your equivalent), and install (you must be root):

<syntaxhighlight lang=Bash>sudo ./install.sh</syntaxhighlight>

Now, go to the directory where you extracted the PrimeSense drivers (<span style="color:#FF8C00">''Sensor-master/''</span> for me), and repeat the exact same procedure (go to <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span>, issue <span style="color:#228B22">''./RedistMaker''</span>, go to <span style="color:#FF8C00">''Platform/Linux/Redist/Sensor-Bin-Linux-x64-v5.1.6.6/''</span>, issue <span style="color:#228B22">''sudo ./install.sh''</span>). Congratulations, you have now installed OpenNI.

====CUDA====

Like OpenNI, [https://en.wikipedia.org/wiki/CUDA nVidia CUDA] is an optional dependency, that will allow PCL to use your GPU (Graphics Processing Unit, that is, your graphics card) for certain computations. This is mandatory for tools like KinFu (do not bother unless you have at least a series 400 card with 1.5 GB of VRAM).

Some distributions provide packages for CUDA in the repositories. For example, in Ubuntu:

<syntaxhighlight lang=Bash>sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit</syntaxhighlight>

If you want to install it manually (which is incompatible with the previous method), go to the [https://developer.nvidia.com/cuda-downloads CUDA downloads] page, which is self-explanatory, and get the small .deb file or the huge .run toolkit/SDK installer file for your system (you should have installed the nVidia drivers already, but the installer will also do it for you if needed).

If you chose the .deb file, install it using the method of your choice, like Gdebi package manager, or through console:

<syntaxhighlight lang=Bash>sudo dpkg -i <package.deb></syntaxhighlight>

The .deb does not contain all CUDA stuff, it just adds their repository to your software lists. Now you must install everything:

<syntaxhighlight lang=Bash>sudo apt-get update
sudo apt-get install cuda</syntaxhighlight>

If, on the other hand, you downloaded the .run, give it execution permissions:

<syntaxhighlight lang=Bash>chmod +x cuda_7.0.28_linux.run</syntaxhighlight>

And install it. You can use the default options, although if you have a working nVidia graphics driver for your card, you may want to say "no" when the installer offers to install it for you:

<syntaxhighlight lang=Bash>sudo ./cuda_7.0.28_linux.run</syntaxhighlight>

The environment variables need to be changed so your system can find CUDA's libraries and binaries. Open <span style="color:#1E90FF">''/etc/ld.so.conf''</span>:

<syntaxhighlight lang=Bash>sudo nano /etc/ld.so.conf</syntaxhighlight>

And append one of these two lines:

<syntaxhighlight lang=Bash>/usr/local/cuda/lib64 # Add this on 64-bit only.
/usr/local/cuda/lib # Add this on 32-bit only.</syntaxhighlight>

Save with <span style="color:#228B22">'''Ctrl+O'''</span> and Enter, exit with <span style="color:#228B22">'''Ctrl+X'''</span>. Reload the cache of the dynamic linker with:

<syntaxhighlight lang=Bash>sudo ldconfig</syntaxhighlight>

Now, append CUDA's bin directory to your PATH. Do this by editing your local <span style="color:#1E90FF">''.bashrc''</span> file:

<syntaxhighlight lang=Bash>nano ~/.bashrc</syntaxhighlight>

And append this line:

<syntaxhighlight lang=Bash>export PATH=$PATH:/usr/local/cuda/bin</syntaxhighlight>

CUDA is now installed.

===Getting the source===

Every dependency is installed. Time to download PCL's source code. First, you must choose whether to install the stable or the experimental branch of PCL. The stable branch is the latest official release and it is guaranteed to work without problems. The experimental branch may give you a compiling error seldomly, but you can find some interesting features that stable users will have to wait some months for. Apart from that, both are built the same way.

To get the stable version, go to the [https://github.com/PointCloudLibrary/pcl/releases downloads] page, get <span style="color:#1E90FF">''pcl-pcl-1.7.2.tar.gz''</span> or whatever the latest release is, and extract it somewhere. For the experimental version, use Git:

<syntaxhighlight lang=Bash>sudo apt-get install git
git clone https://github.com/PointCloudLibrary/pcl PCL-trunk-Source</syntaxhighlight>

The compiled trunk version of PCL will take up more than 8 GB. So make sure you put the source folder in a partition with enough free space!

===Compiling===

Go the the PCL source directory (<span style="color:#FF8C00">''pcl-pcl-1.7.2/''</span> or <span style="color:#FF8C00">''PCL-trunk-Source/''</span>), and create a new subdirectory to keep the build files in:

<syntaxhighlight lang=Bash>mkdir build
cd build</syntaxhighlight>

Now it is time to configure the project using CMake. We will tell it to build in Release (fully optimized, no debug capabilities) mode now, and customize the rest of the options later:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..</syntaxhighlight>

CMake should be able to find every dependency, thus being able to build every subsystem except for the ones marked as <span style="color:#FF1493">''"Disabled by default"''</span>. If you are happy, you can build now, otherwise let's invoke CMake's curses interface to change a couple of things (mind the final dot):

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

[[Image:Ccmake_GUI_PCL.png|thumb|center|900px|Interface of ''ccmake''.]]

Here you can change the build options. The program usage can be found at the bottom of the screen. Try turning all functionality <span style="color:#FF1493">"ON"</span>. The most important thing, in case you want to use CUDA, is to enable it and give CMake the path to your SDK. Go to the <span style="color:#FF1493">"CUDA_SDK_ROOT_DIR"</span> option and enter the correct path (probably <span style="color:#FF8C00">''/usr/local/cuda/''</span> or similar).

When you are done, press <span style="color:#228B22">'''C'''</span> to configure and <span style="color:#228B22">'''G'''</span> to generate and exit the tool. Sometimes, the options you change can activate previously omitted parameters, or prompt some warning text. Just press <span style="color:#228B22">'''E'''</span> when you are finished reading the message, and keep pressing <span style="color:#228B22">'''C'''</span> until it lets you generate (new parameters will be marked with an asterisk, so you can check them and decide whether or not you want further customization).

If you are done configuring, it is time to build:

<syntaxhighlight lang=Bash>make</syntaxhighlight>

<span style="color:#606060">'''''NOTE: Additionally, you can append the parameter -jX to speed up the compilation, X being the number of cores or processors of your PC, plus one.'''''</span>

Remember that, at any time, you can manually force the project to be reconfigured and built from scratch by emptying the <span style="color:#FF8C00">''build/''</span> directory with:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

===Installing===

It will take some time to compile PCL (up to a few hours if your PC is not powerful enough). When it is finished, install it system-wide with:

<syntaxhighlight lang=Bash>sudo make install</syntaxhighlight>

And you should reboot and proceed to the next section, to see if your computer now recognizes (and uses) your Kinect device.

=Testing (OpenNI viewer)=

We are going to write a simple example program that will fetch data from the Kinect or Xtion and present it to the user, using the PCL library. It will also allow to save the current frame (as point cloud) to disk. If you feel lazy, you can download it [http://robotica.unileon.es/~victorm/PCL_openni_viewer.tar.gz here], and skip the next two sections. Otherwise, create a new directory anywhere in your hard disk and proceed.

==CMakeLists.txt==

Inside that directory, create a new text file named <span style="color:#1E90FF">''CMakeLists.txt''</span>. PCL-based programs use the CMake build system, too. Open it with any editor and paste the following content:

<syntaxhighlight lang=CMake>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

project(PCL_openni_viewer)

find_package(PCL 1.7 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

set(PCL_BUILD_TYPE Release)

file(GLOB PCL_openni_viewer_SRC
"src/*.h"
"src/*.cpp"
)
add_executable(openniViewer ${PCL_openni_viewer_SRC})

target_link_libraries (openniViewer ${PCL_LIBRARIES})</syntaxhighlight>

CMake syntax is quite self-explanatory. We ask for a CMake version 2.8 installation, minimum. We declare a new project named <span style="color:#FF1493">"openni_PCL_viewer"</span>. We tell CMake to check for the presence of PCL library development files, version 1.7. If our system can not meet the CMake and PCL version requirement, the process will fail.

Next, we feed the compiler and linker the directories where PCL includes and libraries can be found, and the defined symbols. We tell CMake to use the <span style="color:#FF1493">"Release"</span> build type, which will activate certain optimizations depending on the compiler we use. Other build types are available, like <span style="color:#FF1493">"Debug"</span>, <span style="color:#FF1493">"MinSizeRel"</span>, and <span style="color:#FF1493">"RelWithDebInfo"</span>.

Finally, we create a variable, <span style="color:#FF1493">"opennipclviewer_SRC"</span>, that will store a list of files to be compiled (though we will only have one). We create a new binary to be compiled from these source files, and we link it with the PCL library.

Check the [http://www.cmake.org/Wiki/CMake_Useful_Variables CMake help] for more interesting options.

==main.cpp==

We told CMake it could find the source files in a <span style="color:#FF8C00">''src/''</span> subdirectory, so let's keep to out word and create it. Then, add a new <span style="color:#1E90FF">''main.cpp''</span> file inside and paste the following lines:

<syntaxhighlight lang=CPP>// Original code by Geoffrey Biggs, taken from the PCL tutorial in
// http://pointclouds.org/documentation/tutorials/pcl_visualizer.php

// Simple OpenNI viewer that also allows to write the current scene to a .pcd
// when pressing SPACE.

#include <pcl/io/openni_grabber.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/console/parse.h>

#include <iostream>

using namespace std;
using namespace pcl;

PointCloud<PointXYZRGBA>::Ptr cloudptr(new PointCloud<PointXYZRGBA>); // A cloud that will store color info.
PointCloud<PointXYZ>::Ptr fallbackCloud(new PointCloud<PointXYZ>); // A fallback cloud with just depth data.
boost::shared_ptr<visualization::CloudViewer> viewer; // Point cloud viewer object.
Grabber* openniGrabber; // OpenNI grabber that takes data from the device.
unsigned int filesSaved = 0; // For the numbering of the clouds saved to disk.
bool saveCloud(false), noColor(false); // Program control.

void
printUsage(const char* programName)
{
cout << "Usage: " << programName << " [options]"
<< endl
<< endl
<< "Options:\n"
<< endl
<< "\t<none> start capturing from an OpenNI device.\n"
<< "\t-v FILE visualize the given .pcd file.\n"
<< "\t-h shows this help.\n";
}

// This function is called every time the device has new data.
void
grabberCallback(const PointCloud<PointXYZRGBA>::ConstPtr& cloud)
{
if (! viewer->wasStopped())
viewer->showCloud(cloud);

if (saveCloud)
{
stringstream stream;
stream << "inputCloud" << filesSaved << ".pcd";
string filename = stream.str();
if (io::savePCDFile(filename, *cloud, true) == 0)
{
filesSaved++;
cout << "Saved " << filename << "." << endl;
}
else PCL_ERROR("Problem saving %s.\n", filename.c_str());

saveCloud = false;
}
}

// For detecting when SPACE is pressed.
void
keyboardEventOccurred(const visualization::KeyboardEvent& event,
void* nothing)
{
if (event.getKeySym() == "space" && event.keyDown())
saveCloud = true;
}

// Creates, initializes and returns a new viewer.
boost::shared_ptr<visualization::CloudViewer>
createViewer()
{
boost::shared_ptr<visualization::CloudViewer> v
(new visualization::CloudViewer("OpenNI viewer"));
v->registerKeyboardCallback(keyboardEventOccurred);

return (v);
}

int
main(int argc, char** argv)
{
if (console::find_argument(argc, argv, "-h") >= 0)
{
printUsage(argv[0]);
return -1;
}

bool justVisualize(false);
string filename;
if (console::find_argument(argc, argv, "-v") >= 0)
{
if (argc != 3)
{
printUsage(argv[0]);
return -1;
}

filename = argv[2];
justVisualize = true;
}
else if (argc != 1)
{
printUsage(argv[0]);
return -1;
}

// First mode, open and show a cloud from disk.
if (justVisualize)
{
// Try with color information...
try
{
io::loadPCDFile<PointXYZRGBA>(filename.c_str(), *cloudptr);
}
catch (PCLException e1)
{
try
{
// ...and if it fails, fall back to just depth.
io::loadPCDFile<PointXYZ>(filename.c_str(), *fallbackCloud);
}
catch (PCLException e2)
{
return -1;
}

noColor = true;
}

cout << "Loaded " << filename << "." << endl;
if (noColor)
cout << "This cloud has no RGBA color information present." << endl;
else cout << "This cloud has RGBA color information present." << endl;
}
// Second mode, start fetching and displaying frames from the device.
else
{
openniGrabber = new OpenNIGrabber();
if (openniGrabber == 0)
return -1;
boost::function<void (const PointCloud<PointXYZRGBA>::ConstPtr&)> f =
boost::bind(&grabberCallback, _1);
openniGrabber->registerCallback(f);
}

viewer = createViewer();

if (justVisualize)
{
if (noColor)
viewer->showCloud(fallbackCloud);
else viewer->showCloud(cloudptr);
}
else openniGrabber->start();

// Main loop.
while (! viewer->wasStopped())
boost::this_thread::sleep(boost::posix_time::seconds(1));

if (! justVisualize)
openniGrabber->stop();
}</syntaxhighlight>

Save and close.

==Compiling==

Follow the same steps you used to build PCL. That is, create a new <span style="color:#FF8C00">''build/''</span> subdirectory next to the <span style="color:#FF8C00">''src/''</span> one. Open a terminal there and issue:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..
make</syntaxhighlight>

==Executing==

Still from the same terminal, run the compiled example program:

<syntaxhighlight lang=Bash>./openniViewer</syntaxhighlight>

After some seconds, the main window will appear and the application will start grabbing frames from the device. You can inspect the current point cloud using the mouse, holding the left button to rotate, the right one (or the mouse wheel) to zoom, and the middle one to pan the camera around. At first, you may see only a black screen, or some big colored axes, but no cloud. Try zooming out, to see the whole scene. Another useful key is <span style="color:#228B22">'''R'''</span>, which will reset the camera parameters when pressed. Use it whenever you notice that zooming has gotten slow after some camera movement, or if you still can not see the cloud. See the [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer] tutorial for additional controls and features.

Whenever you feel ready, press the <span style="color:#228B22">'''SPACE'''</span> key. The program will pause for a fraction of a second and the output <span style="color:#FF1493">''"Saved inputCloud0.pcd."''</span> will appear on the console. Check the current folder to see that file <span style="color:#1E90FF">''inputCloud0.pcd''</span> has indeed been written. You can now close the program with <span style="color:#228B22">'''Q'''</span> or <span style="color:#228B22">'''Alt+F4'''</span>.

Next, run it again giving the following parameter:

<syntaxhighlight lang=Bash>./openniViewer -v inputCloud0.pcd</syntaxhighlight>

This will tell the program not to take data from the device, but from the saved point cloud file instead. After it loads, you will realize that you are presented the same scene you saved to disk.

<span style="color:#606060">'''''NOTE: PCD data is saved relative to the sensor. No matter how much you have manipulated the view, it will reset to default when you load the file.'''''</span>

=Conclusion=

At this point, your Kinect device should be working and getting depth data for you. There is a collection of excellent [http://pointclouds.org/documentation/tutorials/ tutorials] for PCL in the official webpage. I encourage you to finish them all before proceeding your experiments with the Kinect sensor. You can also find a good introduction/tutorial at the PCL library [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Introduction.pdf here].

If you use an ASUS Xtion PRO device instead, you should have gotten everything to work without problems or additional steps (except maybe for [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO | this one]]).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 1: Installing and testing

2015-11-01T22:53:30Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

[[Image:Kinect.jpg|thumb|right|200px|Microsoft Kinect device.]]

This series of tutorials will explain the usage of a depth camera like [https://en.wikipedia.org/wiki/Kinect Kinect] for "serious" researching purposes. As you may know, Kinect is in fact an affordable depth sensor, developed with technology from [https://en.wikipedia.org/wiki/PrimeSense PrimeSense], based on infrarred structured light method. It also has a common camera (which makes it a RGB-D device), a microphone and a motorized pivot. Its use is not limited to playing with a Xbox360 console, you can plug it to a computer and use it like any other sensor. Many open-source drivers and frameworks are available.

Since its release on November 2010, it has gained a lot of popularity, specially among the scientific community. Many researches have procured themselves one because, despite the low cost (about 100 $), it has proven to be a powerful solution for depth sensing projects. Current investigations focus on real-time surface mapping, object recognition and tracking, and localization. Impressive results (like the [http://research.microsoft.com/en-us/projects/surfacerecon/ KinectFusion] project from Microsoft) are already possible.

The new [https://en.wikipedia.org/wiki/Xbox_One Xbox One] ships with an upgraded version, [https://en.wikipedia.org/wiki/Kinect_for_Xbox_One Kinect v2], with enhanced resolution, that is able to detect your facial expression, measure your heart rate, and track every one of your fingers. A PC development-ready version ([https://en.wikipedia.org/wiki/Kinect_for_Xbox_One#Kinect_for_Windows Kinect for Windows v2]) was released in July 2014, but it could only be used with the official Windows SDK (open source support [https://github.com/OpenKinect/libfreenect2 exists] but is still young). In October 2014 an adapter that allows to plug the standard Kinect v2 to a PC was released, so the development version of the sensor was discontinued in April 2015. Now you can just buy the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-for-Xbox-One/productID.307499400 standard one] for the console plus the [http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600 adapter].

I will explain the installation and usage of one of the "legacy" Kinect 1.0 devices with a common PC, and the possibilities it offers. I will do it in an easy to understand way, intended for students that have just acquired it and want to start from scratch.

Keep in mind that the software that we are going to use (Point Cloud Library and OpenNI drivers) will also let you use any other device like the [https://www.asus.com/3D-Sensor/Xtion_PRO/ Xtion PRO] or [https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/ Xtion PRO LIVE] from ASUS (the PRO only has a depth sensor, the PRO LIVE is a RGB-D camera) without changing a line of code.

<span style="color:#606060">'''''NOTE: The tutorials are written for Linux platforms. Also, 64-bit versions seem to work better than 32-bit.'''''</span>

=Requirements=

You will need the following:

* A common Kinect device, out of the box. You can buy it in your local electronics shop, or online. It also includes a free copy of ''[https://en.wikipedia.org/wiki/Kinect_Adventures! Kinect Adventures!]'', which is useless if you do not own the console. Microsoft has released a ''Kinect for Windows'' device, which is a normal looking Kinect no longer compatible with Xbox360, that will only work with their official SDK, intended for developers only. Also, like I stated earlier, you can use an ASUS Xtion indistinctly.
* A computer running Linux (Debian or Ubuntu preferably).
* A medium-sized room. Kinect has some limitations for depth measurement: 40cm minimum, 8m maximum (make it 6).

<span style="color:#606060">'''''NOTE: ''Kinect for Windows'' may have problems working with open source drivers on Linux .'''''</span>

=Connecting everything=

Kinect does not work with a common USB port. Its power consumption is a bit higher because of the motor, so Microsoft came up with a connector that combines USB and power supply. Old Xbox 360 models needed a special adapter, new S model ones already have this new port. Luckily, Kinect comes with the official adapter out of the box (otherwise you will have to [http://www.amazon.com/Power-Supply-Cable-Kinect-Xbox-360/dp/B004S7GA46/ref=sr_1_1?ie=UTF8&qid=1378288327&sr=8-1 buy one]).

Just plug the adapter to any power socket, and the USB to your computer. Let's check typing this in a terminal:

<syntaxhighlight lang=Bash>lsusb</syntaxhighlight>

Output should list the following devices:

<syntaxhighlight lang=Bash>Bus 001 Device 005: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 006: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 007: ID 045e:02ae Microsoft Corp. Xbox NUI Camera</syntaxhighlight>

If you are using a Xtion, you should see:

<syntaxhighlight lang=Bash>Bus 001 Device 004: ID 1d27:0601 ASUS</syntaxhighlight>

"0601" identifies the new Xtion model, "0600" the older one. Both should work the same. But try to avoid [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO|USB 3.0 ports]]!

=Installing the software=

There is more than one way to get your Kinect working on your PC, and start developing applications for it:

* [http://www.microsoft.com/en-us/download/details.aspx?id=40278 Kinect for Windows SDK] and [http://www.microsoft.com/en-us/download/details.aspx?id=40276 Developer Toolkit]: released on June 16, 2011 as a non-commercial SDK intended for application development. Version 1.8, the last there will ever be now that Kinect v2 is out, was released on September 2013. Because it comes from Microsoft, it is obviously the easiest way to get everything working. Sadly, there is no Linux version.
* [https://github.com/OpenKinect/libfreenect libfreenect] library: from the [http://openkinect.org/wiki/Main_Page OpenKinect] project, it is intended to be a free and open source alternative to the official drivers. libfreenect is used by projects like [https://github.com/ofTheo/ofxKinect ofxKinect], an addon for the [http://www.openframeworks.cc/ openFrameworks] toolkit (and as of version 0.8, included in the core package) that runs on Linux and OS X. ofxKinect packs a nice example application to show the RGB and point cloud taken from Kinect.
* [https://github.com/PrimeSense/Sensor PrimeSense drivers]: they were released as open source after the the [http://en.wikipedia.org/wiki/OpenNI OpenNI] project was created, along with the motion tracking middleware, ''NITE'', and the SDK. NI stands for Natural Interaction, and the project tried to enforce a common standard for human input using Kinect-like sensors. These official drivers are used by [http://wiki.ros.org/openni_kinect ROS] (the Robot Operating System, a massive collection of libraries and tools for robotic researchers) and [http://pointclouds.org/ PCL] (the Point Cloud Library, with everything needed for 3D point cloud processing). Sadly, version 2.0 of the OpenNI SDK dropped support for Kinect on Linux due to licensing issues, so for now PCL relies on legacy 1.x versions. Also, Apple bought PrimeSense on November 2013, and on April 2014 OpenNI's webpage was closed. The source is now being maintained by [http://structure.io/openni a third party].
* [https://github.com/avin2/SensorKinect SensorKinect]: a modified version of the official PrimeSense drivers, used for example by [https://github.com/gameoverhack/ofxOpenNI ofxOpenNI] (another openFrameworks addon). Last updated on 2012.

For this tutorial, we are going to use PCL with the OpenNI drivers, so owners of a Xtion can also get it to work.

==Precompiled PCL for Ubuntu==

If you have a valid installation of [http://wiki.ros.org/indigo/Installation/Ubuntu ROS] (through their repository), you do not have to install anything. Both OpenNI and PrimeSense drivers, as well as PCL, should be already installed. You can check it with:

<syntaxhighlight lang=Bash>sudo apt-get install libpcl-1.7-all libpcl-1.7-all-dev libopenni-dev libopenni-sensor-primesense-dev</syntaxhighlight>

The previous command should state that all packages are already installed (change the PCL version number as needed), install them if not. If you get an error about overwriting some file, check [[PCL/OpenNI_troubleshooting#Trying_to_overwrite_X.2C_which_is_also_in_package_Y|this]].

In case you do not want ROS, there is a [https://launchpad.net/~v-launchpad-jochen-sprickerhof-de/+archive/ubuntu/pcl PPA] (Personal Package Archive, a private repository) which has everything we need. Add it to your sources, and install everything:

<syntaxhighlight lang=Bash>sudo add-apt-repository ppa:v-launchpad-jochen-sprickerhof-de/pcl
sudo apt-get update
sudo apt-get install build-essential cmake libpcl1.7 libpcl-dev pcl-tools</syntaxhighlight>

Trying to mix ROS and PCL repositories and packages can cause some errors, so choose one of them and stick with it. Check the [[PCL/OpenNI troubleshooting]] page because your Kinect may not work by default in 32 bits.

==Compiling PCL from source==

For Linuxes without a precompiled version of PCL, you will need to compile it yourself. This has an advantage, actually: you can customize the build options and choose what you want. Also, the resulting binaries and libraries should be a bit faster. And you always get the latest version! The instructions are [http://pointclouds.org/documentation/tutorials/compiling_pcl_posix.php here], but the steps are easy so I will show them to you.

===Installing the dependencies===

Some of PCL dependencies can be installed via the package manager. Others will require additional work.

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install build-essential cmake cmake-curses-gui libboost-all-dev libeigen3-dev libflann-dev libvtk5-dev libvtk5-qt4-dev libglew-dev libxmu-dev libsuitesparse-dev libqhull-dev libpcap-dev libxmu-dev libxi-dev libgtest-dev libqt4-opengl-dev</syntaxhighlight>

The trunk version of PCL uses [http://en.wikipedia.org/wiki/VTK VTK] 6 and [http://en.wikipedia.org/wiki/Qt_%28software%29 Qt] 5, so if you intend to compile it, you must install the following packages (say yes if you are asked to remove VTK 5 and Qt 4 packages):

<syntaxhighlight lang=Bash enclose="div">sudo apt-get install libvtk6-dev libqt5opengl5-dev</syntaxhighlight>

====JDK====

OpenNI requires Sun's official JDK (Java Development Kit), which is no longer available on official apt repositories. You can use [http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html unofficial ones], or do it manually. For the last method, go to the [http://www.oracle.com/technetwork/java/javase/downloads/index.html Java SE downloads] page (SE means Standard Edition) and download the latest version (e.g., <span style="color:#1E90FF">''jdk-8u66-linux-x64.tar.gz''</span>). Extract it, then move the contents to <span style="color:#FF8C00">''/usr/lib/jvm/''</span> so it is available system-wide:

<syntaxhighlight lang=Bash>sudo mkdir -p /usr/lib/jvm/
sudo cp -r jdk1.8.0_66/ /usr/lib/jvm/</syntaxhighlight>

Then, make it the default choice to compile and run Java programs. Remember to change the version number as needed!

<syntaxhighlight lang=Bash>sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_66/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_66/bin/javac" 1
sudo update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib/jvm/jdk1.8.0_66/bin/jar" 1</syntaxhighlight>

To be sure, use the following commands to manually select the correct option, in case there is more than one choice:

<syntaxhighlight lang=Bash>sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config jar</syntaxhighlight>

If you are still not sure, run them and display the version, making sure it is the one you installed:

<syntaxhighlight lang=Bash>java -version
javac -version</syntaxhighlight>

Sun's JDK is now installed.

====OpenNI====

PCL uses OpenNI and the PrimeSense drivers to get data from devices like Kinect or Xtion. It is optional, but it would not make much sense not to install it, would it? If you are using Ubuntu, add the [[#Precompiled PCL for Ubuntu | PPA]] and install <span style="color:#228B22">''libopenni-dev''</span> and <span style="color:#228B22">''libopenni-sensor-primesense-dev''</span>, which should be done already. Otherwise, fetch the [https://github.com/OpenNI/OpenNI OpenNI] and [https://github.com/PrimeSense/Sensor PrimeSense Sensor] sources from GitHub (download them as .zip, the link is on the right). Extract them, and install the dependencies:

<syntaxhighlight lang=Bash>sudo apt-get install python libusb-1.0-0-dev freeglut3-dev doxygen graphviz</syntaxhighlight>

Go to the directory where you extracted OpenNI (<span style="color:#FF8C00">''OpenNI-master/''</span> for me), and open a terminal in the <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span> subdirectory. Issue:

<syntaxhighlight lang=Bash>./RedistMaker</syntaxhighlight>

When it finishes, and if there are no errors (check the [[PCL/OpenNI troubleshooting]] page if you get any), go to <span style="color:#FF8C00">''Platform/Linux/Redist/OpenNI-Bin-Dev-Linux-x64-v1.5.7.10/''</span> (or your equivalent), and install (you must be root):

<syntaxhighlight lang=Bash>sudo ./install.sh</syntaxhighlight>

Now, go to the directory where you extracted the PrimeSense drivers (<span style="color:#FF8C00">''Sensor-master/''</span> for me), and repeat the exact same procedure (go to <span style="color:#FF8C00">''Platform/Linux/CreateRedist/''</span>, issue <span style="color:#228B22">''./RedistMaker''</span>, go to <span style="color:#FF8C00">''Platform/Linux/Redist/Sensor-Bin-Linux-x64-v5.1.6.6/''</span>, issue <span style="color:#228B22">''sudo ./install.sh''</span>). Congratulations, you have now installed OpenNI.

====CUDA====

Like OpenNI, [http://en.wikipedia.org/wiki/CUDA nVidia CUDA] is an optional dependency, that will allow PCL to use your GPU (Graphics Processing Unit, that is, your graphics card) for certain computations. This is mandatory for tools like KinFu (do not bother unless you have at least a series 400 card with 1.5 GB of VRAM).

Some distributions provide packages for CUDA in the repositories. For example, in Ubuntu:

<syntaxhighlight lang=Bash>sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit</syntaxhighlight>

If you want to install it manually (which is incompatible with the previous method), go to the [https://developer.nvidia.com/cuda-downloads CUDA downloads] page, which is self-explanatory, and get the small .deb file or the huge .run toolkit/SDK installer file for your system (you should have installed the nVidia drivers already, but the installer will also do it for you if needed).

If you chose the .deb file, install it using the method of your choice, like Gdebi package manager, or through console:

<syntaxhighlight lang=Bash>sudo dpkg -i <package.deb></syntaxhighlight>

The .deb does not contain all CUDA stuff, it just adds their repository to your software lists. Now you must install everything:

<syntaxhighlight lang=Bash>sudo apt-get update
sudo apt-get install cuda</syntaxhighlight>

If, on the other hand, you downloaded the .run, give it execution permissions:

<syntaxhighlight lang=Bash>chmod +x cuda_7.0.28_linux.run</syntaxhighlight>

And install it. You can use the default options, although if you have a working nVidia graphics driver for your card, you may want to say "no" when the installer offers to install it for you:

<syntaxhighlight lang=Bash>sudo ./cuda_7.0.28_linux.run</syntaxhighlight>

The environment variables need to be changed so your system can find CUDA's libraries and binaries. Open <span style="color:#1E90FF">''/etc/ld.so.conf''</span>:

<syntaxhighlight lang=Bash>sudo nano /etc/ld.so.conf</syntaxhighlight>

And append one of these two lines:

<syntaxhighlight lang=Bash>/usr/local/cuda/lib64 # Add this on 64-bit only.
/usr/local/cuda/lib # Add this on 32-bit only.</syntaxhighlight>

Save with <span style="color:#228B22">'''Ctrl+O'''</span> and Enter, exit with <span style="color:#228B22">'''Ctrl+X'''</span>. Reload the cache of the dynamic linker with:

<syntaxhighlight lang=Bash>sudo ldconfig</syntaxhighlight>

Now, append CUDA's bin directory to your PATH. Do this by editing your local <span style="color:#1E90FF">''.bashrc''</span> file:

<syntaxhighlight lang=Bash>nano ~/.bashrc</syntaxhighlight>

And append this line:

<syntaxhighlight lang=Bash>export PATH=$PATH:/usr/local/cuda/bin</syntaxhighlight>

CUDA is now installed.

===Getting the source===

Every dependency is installed. Time to download PCL's source code. First, you must choose whether to install the stable or the experimental branch of PCL. The stable branch is the latest official release and it is guaranteed to work without problems. The experimental branch may give you a compiling error seldomly, but you can find some interesting features that stable users will have to wait some months for. Apart from that, both are built the same way.

To get the stable version, go to the [https://github.com/PointCloudLibrary/pcl/releases downloads] page, get <span style="color:#1E90FF">''pcl-pcl-1.7.2.tar.gz''</span> or whatever the latest release is, and extract it somewhere. For the experimental version, use Git:

<syntaxhighlight lang=Bash>sudo apt-get install git
git clone https://github.com/PointCloudLibrary/pcl PCL-trunk-Source</syntaxhighlight>

The compiled trunk version of PCL will take up more than 8 GB. So make sure you put the source folder in a partition with enough free space!

===Compiling===

Go the the PCL source directory (<span style="color:#FF8C00">''pcl-pcl-1.7.2/''</span> or <span style="color:#FF8C00">''PCL-trunk-Source/''</span>), and create a new subdirectory to keep the build files in:

<syntaxhighlight lang=Bash>mkdir build
cd build</syntaxhighlight>

Now it is time to configure the project using CMake. We will tell it to build in Release (fully optimized, no debug capabilities) mode now, and customize the rest of the options later:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..</syntaxhighlight>

CMake should be able to find every dependency, thus being able to build every subsystem except for the ones marked as <span style="color:#FF1493">''"Disabled by default"''</span>. If you are happy, you can build now, otherwise let's invoke CMake's curses interface to change a couple of things (mind the final dot):

<syntaxhighlight lang=Bash>ccmake .</syntaxhighlight>

[[Image:Ccmake_GUI_PCL.png|thumb|center|900px|Interface of ''ccmake''.]]

Here you can change the build options. The program usage can be found at the bottom of the screen. Try turning all functionality <span style="color:#FF1493">"ON"</span>. The most important thing, in case you want to use CUDA, is to enable it and give CMake the path to your SDK. Go to the <span style="color:#FF1493">"CUDA_SDK_ROOT_DIR"</span> option and enter the correct path (probably <span style="color:#FF8C00">''/usr/local/cuda/''</span> or similar).

When you are done, press <span style="color:#228B22">'''C'''</span> to configure and <span style="color:#228B22">'''G'''</span> to generate and exit the tool. Sometimes, the options you change can activate previously omitted parameters, or prompt some warning text. Just press <span style="color:#228B22">'''E'''</span> when you are finished reading the message, and keep pressing <span style="color:#228B22">'''C'''</span> until it lets you generate (new parameters will be marked with an asterisk, so you can check them and decide whether or not you want further customization).

If you are done configuring, it is time to build:

<syntaxhighlight lang=Bash>make</syntaxhighlight>

<span style="color:#606060">'''''NOTE: Additionally, you can append the parameter -jX to speed up the compilation, X being the number of cores or processors of your PC, plus one.'''''</span>

Remember that, at any time, you can manually force the project to be reconfigured and built from scratch by emptying the <span style="color:#FF8C00">''build/''</span> directory with:

<syntaxhighlight lang=Bash>rm -rf ./*</syntaxhighlight>

===Installing===

It will take some time to compile PCL (up to a few hours if your PC is not powerful enough). When it is finished, install it system-wide with:

<syntaxhighlight lang=Bash>sudo make install</syntaxhighlight>

And you should reboot and proceed to the next section, to see if your computer now recognizes (and uses) your Kinect device.

=Testing (OpenNI viewer)=

We are going to write a simple example program that will fetch data from the Kinect or Xtion and present it to the user, using the PCL library. It will also allow to save the current frame (as point cloud) to disk. If you feel lazy, you can download it [http://robotica.unileon.es/~victorm/PCL_openni_viewer.tar.gz here], and skip the next two sections. Otherwise, create a new directory anywhere in your hard disk and proceed.

==CMakeLists.txt==

Inside that directory, create a new text file named <span style="color:#1E90FF">''CMakeLists.txt''</span>. PCL-based programs use the CMake build system, too. Open it with any editor and paste the following content:

<syntaxhighlight lang=CMake>cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

project(PCL_openni_viewer)

find_package(PCL 1.7 REQUIRED)

include_directories(${PCL_INCLUDE_DIRS})
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

set(PCL_BUILD_TYPE Release)

file(GLOB PCL_openni_viewer_SRC
"src/*.h"
"src/*.cpp"
)
add_executable(openniViewer ${PCL_openni_viewer_SRC})

target_link_libraries (openniViewer ${PCL_LIBRARIES})</syntaxhighlight>

CMake syntax is quite self-explanatory. We ask for a CMake version 2.8 installation, minimum. We declare a new project named <span style="color:#FF1493">"openni_PCL_viewer"</span>. We tell CMake to check for the presence of PCL library development files, version 1.7. If our system can not meet the CMake and PCL version requirement, the process will fail.

Next, we feed the compiler and linker the directories where PCL includes and libraries can be found, and the defined symbols. We tell CMake to use the <span style="color:#FF1493">"Release"</span> build type, which will activate certain optimizations depending on the compiler we use. Other build types are available, like <span style="color:#FF1493">"Debug"</span>, <span style="color:#FF1493">"MinSizeRel"</span>, and <span style="color:#FF1493">"RelWithDebInfo"</span>.

Finally, we create a variable, <span style="color:#FF1493">"opennipclviewer_SRC"</span>, that will store a list of files to be compiled (though we will only have one). We create a new binary to be compiled from these source files, and we link it with the PCL library.

Check the [http://www.cmake.org/Wiki/CMake_Useful_Variables CMake help] for more interesting options.

==main.cpp==

We told CMake it could find the source files in a <span style="color:#FF8C00">''src/''</span> subdirectory, so let's keep to out word and create it. Then, add a new <span style="color:#1E90FF">''main.cpp''</span> file inside and paste the following lines:

<syntaxhighlight lang=CPP>// Original code by Geoffrey Biggs, taken from the PCL tutorial in
// http://pointclouds.org/documentation/tutorials/pcl_visualizer.php

// Simple OpenNI viewer that also allows to write the current scene to a .pcd
// when pressing SPACE.

#include <pcl/io/openni_grabber.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/console/parse.h>

#include <iostream>

using namespace std;
using namespace pcl;

PointCloud<PointXYZRGBA>::Ptr cloudptr(new PointCloud<PointXYZRGBA>); // A cloud that will store color info.
PointCloud<PointXYZ>::Ptr fallbackCloud(new PointCloud<PointXYZ>); // A fallback cloud with just depth data.
boost::shared_ptr<visualization::CloudViewer> viewer; // Point cloud viewer object.
Grabber* openniGrabber; // OpenNI grabber that takes data from the device.
unsigned int filesSaved = 0; // For the numbering of the clouds saved to disk.
bool saveCloud(false), noColor(false); // Program control.

void
printUsage(const char* programName)
{
cout << "Usage: " << programName << " [options]"
<< endl
<< endl
<< "Options:\n"
<< endl
<< "\t<none> start capturing from an OpenNI device.\n"
<< "\t-v FILE visualize the given .pcd file.\n"
<< "\t-h shows this help.\n";
}

// This function is called every time the device has new data.
void
grabberCallback(const PointCloud<PointXYZRGBA>::ConstPtr& cloud)
{
if (! viewer->wasStopped())
viewer->showCloud(cloud);

if (saveCloud)
{
stringstream stream;
stream << "inputCloud" << filesSaved << ".pcd";
string filename = stream.str();
if (io::savePCDFile(filename, *cloud, true) == 0)
{
filesSaved++;
cout << "Saved " << filename << "." << endl;
}
else PCL_ERROR("Problem saving %s.\n", filename.c_str());

saveCloud = false;
}
}

// For detecting when SPACE is pressed.
void
keyboardEventOccurred(const visualization::KeyboardEvent& event,
void* nothing)
{
if (event.getKeySym() == "space" && event.keyDown())
saveCloud = true;
}

// Creates, initializes and returns a new viewer.
boost::shared_ptr<visualization::CloudViewer>
createViewer()
{
boost::shared_ptr<visualization::CloudViewer> v
(new visualization::CloudViewer("OpenNI viewer"));
v->registerKeyboardCallback(keyboardEventOccurred);

return (v);
}

int
main(int argc, char** argv)
{
if (console::find_argument(argc, argv, "-h") >= 0)
{
printUsage(argv[0]);
return -1;
}

bool justVisualize(false);
string filename;
if (console::find_argument(argc, argv, "-v") >= 0)
{
if (argc != 3)
{
printUsage(argv[0]);
return -1;
}

filename = argv[2];
justVisualize = true;
}
else if (argc != 1)
{
printUsage(argv[0]);
return -1;
}

// First mode, open and show a cloud from disk.
if (justVisualize)
{
// Try with color information...
try
{
io::loadPCDFile<PointXYZRGBA>(filename.c_str(), *cloudptr);
}
catch (PCLException e1)
{
try
{
// ...and if it fails, fall back to just depth.
io::loadPCDFile<PointXYZ>(filename.c_str(), *fallbackCloud);
}
catch (PCLException e2)
{
return -1;
}

noColor = true;
}

cout << "Loaded " << filename << "." << endl;
if (noColor)
cout << "This cloud has no RGBA color information present." << endl;
else cout << "This cloud has RGBA color information present." << endl;
}
// Second mode, start fetching and displaying frames from the device.
else
{
openniGrabber = new OpenNIGrabber();
if (openniGrabber == 0)
return -1;
boost::function<void (const PointCloud<PointXYZRGBA>::ConstPtr&)> f =
boost::bind(&grabberCallback, _1);
openniGrabber->registerCallback(f);
}

viewer = createViewer();

if (justVisualize)
{
if (noColor)
viewer->showCloud(fallbackCloud);
else viewer->showCloud(cloudptr);
}
else openniGrabber->start();

// Main loop.
while (! viewer->wasStopped())
boost::this_thread::sleep(boost::posix_time::seconds(1));

if (! justVisualize)
openniGrabber->stop();
}</syntaxhighlight>

Save and close.

==Compiling==

Follow the same steps you used to build PCL. That is, create a new <span style="color:#FF8C00">''build/''</span> subdirectory next to the <span style="color:#FF8C00">''src/''</span> one. Open a terminal there and issue:

<syntaxhighlight lang=Bash>cmake -DCMAKE_BUILD_TYPE=Release ..
make</syntaxhighlight>

==Executing==

Still from the same terminal, run the compiled example program:

<syntaxhighlight lang=Bash>./openniViewer</syntaxhighlight>

After some seconds, the main window will appear and the application will start grabbing frames from the device. You can inspect the current point cloud using the mouse, holding the left button to rotate, the right one (or the mouse wheel) to zoom, and the middle one to pan the camera around. At first, you may see only a black screen, or some big colored axes, but no cloud. Try zooming out, to see the whole scene. Another useful key is <span style="color:#228B22">'''R'''</span>, which will reset the camera parameters when pressed. Use it whenever you notice that zooming has gotten slow after some camera movement, or if you still can not see the cloud. See the [http://pointclouds.org/documentation/tutorials/pcl_visualizer.php PCLVisualizer] tutorial for additional controls and features.

Whenever you feel ready, press the <span style="color:#228B22">'''SPACE'''</span> key. The program will pause for a fraction of a second and the output <span style="color:#FF1493">''"Saved inputCloud0.pcd."''</span> will appear on the console. Check the current folder to see that file <span style="color:#1E90FF">''inputCloud0.pcd''</span> has indeed been written. You can now close the program with <span style="color:#228B22">'''Q'''</span> or <span style="color:#228B22">'''Alt+F4'''</span>.

Next, run it again giving the following parameter:

<syntaxhighlight lang=Bash>./openniViewer -v inputCloud0.pcd</syntaxhighlight>

This will tell the program not to take data from the device, but from the saved point cloud file instead. After it loads, you will realize that you are presented the same scene you saved to disk.

<span style="color:#606060">'''''NOTE: PCD data is saved relative to the sensor. No matter how much you have manipulated the view, it will reset to default when you load the file.'''''</span>

=Conclusion=

At this point, your Kinect device should be working and getting depth data for you. There is a collection of excellent [http://pointclouds.org/documentation/tutorials/ tutorials] for PCL in the official webpage. I encourage you to finish them all before proceeding your experiments with the Kinect sensor. You can also find a good introduction/tutorial at the PCL library [http://www.pointclouds.org/assets/uploads/PCL-IAS13_Introduction.pdf here].

If you use an ASUS Xtion PRO device instead, you should have gotten everything to work without problems or additional steps (except maybe for [[PCL/OpenNI_troubleshooting#ASUS_Xtion_PRO | this one]]).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 0: The very basics

2015-11-01T22:50:37Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

One of the most important fields of robotics is [https://en.wikipedia.org/wiki/Computer_vision computer vision]. A machine that is able to "see" its environment can have a lot of applications. For example, think of a robot in a production line that grabs some parts and moves them somewhere else. Or a surveillance system that is able to recognize how many people are there in a room. Or a biped robot making its way through a room, evading obstacles such as tables, chairs or bystanders.

For many years, the most common sensors for computer vision were 2D cameras, that retrieved a RGB image of the scene (like all the digital cameras that are so common nowadays, in our laptops or smarphones). Algorithms exist that are able to find an object in a picture, even if it is rotated or scaled, or that can retrieve motion data from a stream of video, and even perform 3D analysis to track the camera's position. All these years' worth of research is now available in libraries like [https://en.wikipedia.org/wiki/OpenCV OpenCV] (Open Source Computer Vision Library).

3D sensors are available, too. The biggest advantage they offer is that it is trivial to get measurements about distances and motion, but the addition of a new dimension makes calculations expensive. Working with the data they retrieve is a lot different that working with a 2D image, and texture information is rarely used.

During the next tutorials I will explain how to get a common depth sensor working with a 3D processing library.

=Depth sensors=

3D or depth sensors give you precise information about the distance to the "points" in the scene (a point would be the 3D equivalent of a pixel). There are several types of depth sensors, each working with a different technique. Some sensors are slow, some are fast. Some give accurate, high-res measurements, some are noisy and low-res. Some are expensive, some can be bought for a hundred bucks. There is no "perfect" sensor and which one you choose will depend on your budget and the project that you want to implement.

==Stereo cameras==

[https://en.wikipedia.org/wiki/Computer_stereo_vision Stereo cameras] are the only passive measurement device of the list. They are essentially two identical cameras assembled together (some centimeters apart), that capture slightly different scenes. By computing the differences between both scenes, it is possible to infer depth information about the points in each image.

A stereo pair is cheap, but perhaps the least accurate sensor. Ideally, it would require perfect calibration of both cameras, which is unfeasible in practice. Bad light conditions will render it useless. Also, because of the way the algorithm works (detecting corresponding points of interest in both images), they give poor results with empty scenes or objects that have plain textures, with few interest points. Some models circumvent this by projecting a grid or texture of light on the scene (active stereo vision).

I will not go into detail about the math involved with the triangulation process, as you can find it on the internet.

[[Image:Stereo.png|thumb|center|200px|Example of stereo camera.]]

==Time-of-flight==

[https://en.wikipedia.org/wiki/Time_of_flight Time-of-flight] (ToF) sensors work by measuring the time it has taken a ray or pulse of light to travel a certain distance. Because the speed of light is a known constant, a simple formula can be used to obtain the range to the object. These sensors are not affected by light conditions and have the potential to be very precise.

===LIDAR===

A [https://en.wikipedia.org/wiki/Lidar LIDAR] (light+radar) is just a common laser range finder mounted on a platform that is able to rotate very fast, scanning the scene point by point. They are very precise sensors, but also expensive, and they do not retrieve texture information. They have been used for decades in many different fields like meteorology, archaeology or astronomy. LIDAR devices can be mounted on satellites, planes or mobile robots. The data retrieved by a LIDAR has very high resolution, so some processing is needed in order to use it for real-time applications.

[[Image:LIDAR.jpg|thumb|center|200px|LIDAR mounted on a mobile robot (image from Wikimedia Commons).]]

===ToF cameras===

A [https://en.wikipedia.org/wiki/Time-of-flight_camera time-of-flight camera] does not perform point-by-point scans like a LIDAR does. Instead, it employs a single pulse of light to capture the whole scene, once per frame. Thanks to that, they can work a lot faster, with some models topping above 100 Hz. The price to pay is a low resolution of 320×240 or even less. Depth measurements have an accuracy of about 1cm. ToF cameras cost less than LIDAR sensors, but we are talking about 4000 $ or so. Color information is not retrieved.

The new [https://dev.windows.com/en-us/kinect Kinect v2] is a ToF sensor that works at 512×424 internally, and it includes a 1920x1080 RGB camera.

<center><gallery widths=300px>
File:ToFcam.png | Time-of-flight camera (image from Wikimedia Commons).
File:Kinect2.png | Microsoft Kinect v2.
</gallery></center>

==Structured light==

[https://en.wikipedia.org/wiki/Structured-light_3D_scanner Structured light] sensors (like Kinect and Xtion) work by projecting a pattern of infrarred light (for example, a grid of lines, or a "constellation" of points) on top of the scene's objects. This pattern is seen distorted when looked from a perspective different from the projector's. By analysing this distortion, information about the depth can be retrieved, and the surface(s) reconstructed.

The resolution and speed of these sensors are similar to common VGA cameras, usually 640x480 at 30 FPS (actually less because of the way this type of sensors work, Kinect uses 320x240 internally, the rest of the data is extrapolated to "fill the gaps"). Precision is similar to that of ToF cameras, 1cm more or less, with a maximum range of 3-6m, but they tend to have trouble seeing small objects. They are a lot cheaper than any other sensors, with a first generation Kinect now costing around 100 $, and hence they have become a lot popular in the last years. In the next tutorial, I will explain the installation and usage of one of these cameras.

<center><gallery widths=300px>
File:Kinect.jpg | Microsoft Kinect.
File:Xtion.jpg | ASUS Xtion PRO (the PRO LIVE model includes a RGB camera).
</gallery></center>

[[Image:Structured_light.png|thumb|center|200px|Infrarred light pattern (dots) projected by Kinect.]]

=Point clouds=

[[Image:Point_cloud.png|thumb|right|200px|Example of a point cloud.]]

Depth sensors return 3D data in the form of a [https://en.wikipedia.org/wiki/Point_cloud ''point cloud'']. A point cloud is a set of points in three-dimensional space, each with its own XYZ coordinates. In the case of stereo, ToF or structured light cameras, every point corresponds to exactly one pixel of the captured image. Optionally, points can store additional information, like color (if the sensor has a RGB camera that it uses to put texture on them). If you have background with 3D modelling software, then you must know that point clouds can be converted to triangular meshes with certain algorithms, but this is rarely done, and cloud processing is performed with the original set of data.

==Point Cloud Library==

[https://en.wikipedia.org/wiki/Point_Cloud_Library Point Cloud Library (PCL)] is a project started in early 2010 by [https://en.wikipedia.org/wiki/Willow_Garage Willow Garage], the same robotics research company behind the [https://en.wikipedia.org/wiki/Robot_Operating_System Robot Operating System (ROS)] and OpenCV (and they also sell [http://www.willowgarage.com/robot/overview robots] like TurtleBot, too). The first version was [http://www.pointclouds.org/assets/pdf/pcl_icra2011.pdf fully introduced] in 2011, and it has been actively maintained ever since.

PCL aims to be an one-for-all solution for point cloud and 3D processing. It is an open source, multiplatform library divided in many [http://pointclouds.org/documentation/tutorials/walkthrough.php submodules] for different tasks, like visualization, filtering, segmentation, registration, searching, feature estimation... Additionally, it can be used to retrieve data from a list of compatible sensors, so it is an obvious choice for our tutorials, because you will not have to compile or install dozens of separate libraries.

Apart from Willow Garage, PCL is backed by [http://www.openperception.org/ Open Perception], a non-profit organization founded by Radu Bogdan Rusu, one of the most active developers of the library. I recommend you to read his very interesting [http://files.rbrusu.com/publications/RusuPhDThesis.pdf dissertation], written in 2009, which introduces and details many concepts and techniques that have been implemented into PCL.

=Conclusion=

Over the course of the next tutorials I will tell you how to install PCL in a Linux environment, how to retrieve point clouds from a sensor like Kinect or Xtion, and how to work with them for things like object recognition. I will keep it nice and easy, just like I want tutorials to be, and leave complex stuff aside (you can always search the original paper if you want to learn more about some technique).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 0: The very basics

2015-11-01T22:48:34Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

One of the most important fields of robotics is [https://en.wikipedia.org/wiki/Computer_vision computer vision]. A machine that is able to "see" its environment can have a lot of applications. For example, think of a robot in a production line that grabs some parts and moves them somewhere else. Or a surveillance system that is able to recognize how many people are there in a room. Or a biped robot making its way through a room, evading obstacles such as tables, chairs or bystanders.

For many years, the most common sensors for computer vision were 2D cameras, that retrieved a RGB image of the scene (like all the digital cameras that are so common nowadays, in our laptops or smarphones). Algorithms exist that are able to find an object in a picture, even if it is rotated or scaled, or that can retrieve motion data from a stream of video, and even perform 3D analysis to track the camera's position. All these years' worth of research is now available in libraries like [https://en.wikipedia.org/wiki/OpenCV OpenCV] (Open Source Computer Vision Library).

3D sensors are available, too. The biggest advantage they offer is that it is trivial to get measurements about distances and motion, but the addition of a new dimension makes calculations expensive. Working with the data they retrieve is a lot different that working with a 2D image, and texture information is rarely used.

During the next tutorials I will explain how to get a common depth sensor working with a 3D processing library.

=Depth sensors=

3D or depth sensors give you precise information about the distance to the "points" in the scene (a point would be the 3D equivalent of a pixel). There are several types of depth sensors, each working with a different technique. Some sensors are slow, some are fast. Some give accurate, high-res measurements, some are noisy and low-res. Some are expensive, some can be bought for a hundred bucks. There is no "perfect" sensor and which one you choose will depend on your budget and the project that you want to implement.

==Stereo cameras==

[https://en.wikipedia.org/wiki/Computer_stereo_vision Stereo cameras] are the only passive measurement device of the list. They are essentially two identical cameras assembled together (some centimeters apart), that capture slightly different scenes. By computing the differences between both scenes, it is possible to infer depth information about the points in each image.

A stereo pair is cheap, but perhaps the least accurate sensor. Ideally, it would require perfect calibration of both cameras, which is unfeasible in practice. Bad light conditions will render it useless. Also, because of the way the algorithm works (detecting corresponding points of interest in both images), they give poor results with empty scenes or objects that have plain textures, with few interest points. Some models circumvent this by projecting a grid or texture of light on the scene (active stereo vision).

I will not go into detail about the math involved with the triangulation process, as you can find it on the internet.

[[Image:Stereo.png|thumb|center|200px|Example of stereo camera.]]

==Time-of-flight==

[https://en.wikipedia.org/wiki/Time_of_flight Time-of-flight] (ToF) sensors work by measuring the time it has taken a ray or pulse of light to travel a certain distance. Because the speed of light is a known constant, a simple formula can be used to obtain the range to the object. These sensors are not affected by light conditions and have the potential to be very precise.

===LIDAR===

A [https://en.wikipedia.org/wiki/Lidar LIDAR] (light+radar) is just a common laser range finder mounted on a platform that is able to rotate very fast, scanning the scene point by point. They are very precise sensors, but also expensive, and they do not retrieve texture information. They have been used for decades in many different fields like meteorology, archaeology or astronomy. LIDAR devices can be mounted on satellites, planes or mobile robots. The data retrieved by a LIDAR has very high resolution, so some processing is needed in order to use it for real-time applications.

[[Image:LIDAR.jpg|thumb|center|200px|LIDAR mounted on a mobile robot (image from Wikimedia Commons).]]

===ToF cameras===

A [https://en.wikipedia.org/wiki/Time-of-flight_camera time-of-flight camera] does not perform point-by-point scans like a LIDAR does. Instead, it employs a single pulse of light to capture the whole scene, once per frame. Thanks to that, they can work a lot faster, with some models topping above 100 Hz. The price to pay is a low resolution of 320×240 or even less. Depth measurements have an accuracy of about 1cm. ToF cameras cost less than LIDAR sensors, but we are talking about 4000 $ or so. Color information is not retrieved.

The new [https://dev.windows.com/en-us/kinect Kinect v2] is a ToF sensor that works at 512×424 internally, and it includes a 1920x1080 RGB camera.

<center><gallery widths=300px>
File:ToFcam.png | Time-of-flight camera (image from Wikimedia Commons).
File:Kinect2.png | Microsoft Kinect v2.
</gallery></center>

==Structured light==

[https://en.wikipedia.org/wiki/Structured-light_3D_scanner Structured light] sensors (like Kinect and Xtion) work by projecting a pattern of infrarred light (for example, a grid of lines, or a "constellation" of points) on top of the scene's objects. This pattern is seen distorted when looked from a perspective different from the projector's. By analysing this distortion, information about the depth can be retrieved, and the surface(s) reconstructed.

The resolution and speed of these sensors are similar to common VGA cameras, usually 640x480 at 30 FPS (actually less because of the way this type of sensors work, Kinect uses 320x240 internally, the rest of the data is extrapolated to "fill the gaps"). Precision is similar to that of ToF cameras, 1cm more or less, with a maximum range of 3-6m, but they tend to have trouble seeing small objects. They are a lot cheaper than any other sensors, with a first generation Kinect now costing around 100 $, and hence they have become a lot popular in the last years. In the next tutorial, I will explain the installation and usage of one of these cameras.

<center><gallery widths=300px>
File:Kinect.jpg | Microsoft Kinect.
File:Xtion.jpg | ASUS Xtion PRO (the PRO LIVE model includes a RGB camera).
</gallery></center>

[[Image:Structured_light.png|thumb|center|200px|Infrarred light pattern (dots) projected by Kinect.]]

=Point clouds=

[[Image:Point_cloud.png|thumb|right|200px|Example of a point cloud.]]

Depth sensors return 3D data in the form of a [http://en.wikipedia.org/wiki/Point_cloud ''point cloud'']. A point cloud is a set of points in three-dimensional space, each with its own XYZ coordinates. In the case of stereo, ToF or structured light cameras, every point corresponds to exactly one pixel of the captured image. Optionally, points can store additional information, like color (if the sensor has a RGB camera that it uses to put texture on them). If you have background with 3D modelling software, then you must know that point clouds can be converted to triangular meshes with certain algorithms, but this is rarely done, and cloud processing is performed with the original set of data.

==Point Cloud Library==

[http://www.pointclouds.org/ Point Cloud Library (PCL)] is a project started in early 2010 by [http://www.willowgarage.com/ Willow Garage], the same robotics research company behind the [http://www.ros.org/ Robot Operating System (ROS)] and OpenCV (and they also sell [http://www.willowgarage.com/robot/overview robots] like TurtleBot, too). The first version was [http://www.pointclouds.org/assets/pdf/pcl_icra2011.pdf fully introduced] in 2011, and it has been actively maintained ever since.

PCL aims to be an one-for-all solution for point cloud and 3D processing. It is an open source, multiplatform library divided in many [http://pointclouds.org/documentation/tutorials/walkthrough.php submodules] for different tasks, like visualization, filtering, segmentation, registration, searching, feature estimation... Additionally, it can be used to retrieve data from a list of compatible sensors, so it is an obvious choice for our tutorials, because you will not have to compile or install dozens of separate libraries.

Apart from Willow Garage, PCL is backed by [http://www.openperception.org/ Open Perception], a non-profit organization founded by Radu Bogdan Rusu, one of the most active developers of the library. I recommend you to read his very interesting [http://files.rbrusu.com/publications/RusuPhDThesis.pdf dissertation], written in 2009, which introduces and details many concepts and techniques that have been implemented into PCL.

=Conclusion=

Over the course of the next tutorials I will tell you how to install PCL in a Linux environment, how to retrieve point clouds from a sensor like Kinect or Xtion, and how to work with them for things like object recognition. I will keep it nice and easy, just like I want tutorials to be, and leave complex stuff aside (you can always search the original paper if you want to learn more about some technique).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]

PCL/OpenNI tutorial 0: The very basics

2015-11-01T22:45:05Z

Victorm:

Go to root: [[PhD-3D-Object-Tracking]]

----
----

One of the most important fields of robotics is [https://en.wikipedia.org/wiki/Computer_vision computer vision]. A machine that is able to "see" its environment can have a lot of applications. For example, think of a robot in a production line that grabs some parts and moves them somewhere else. Or a surveillance system that is able to recognize how many people are there in a room. Or a biped robot making its way through a room, evading obstacles such as tables, chairs or bystanders.

For many years, the most common sensors for computer vision were 2D cameras, that retrieved a RGB image of the scene (like all the digital cameras that are so common nowadays, in our laptops or smarphones). Algorithms exist that are able to find an object in a picture, even if it is rotated or scaled, or that can retrieve motion data from a stream of video, and even perform 3D analysis to track the camera's position. All these years' worth of research is now available in libraries like [https://en.wikipedia.org/wiki/OpenCV OpenCV] (Open Source Computer Vision Library).

3D sensors are available, too. The biggest advantage they offer is that it is trivial to get measurements about distances and motion, but the addition of a new dimension makes calculations expensive. Working with the data they retrieve is a lot different that working with a 2D image, and texture information is rarely used.

During the next tutorials I will explain how to get a common depth sensor working with a 3D processing library.

=Depth sensors=

3D or depth sensors give you precise information about the distance to the "points" in the scene (a point would be the 3D equivalent of a pixel). There are several types of depth sensors, each working with a different technique. Some sensors are slow, some are fast. Some give accurate, high-res measurements, some are noisy and low-res. Some are expensive, some can be bought for a hundred bucks. There is no "perfect" sensor and which one you choose will depend on your budget and the project that you want to implement.

==Stereo cameras==

[http://en.wikipedia.org/wiki/Computer_stereo_vision Stereo cameras] are the only passive measurement device of the list. They are essentially two identical cameras assembled together (some centimeters apart), that capture slightly different scenes. By computing the differences between both scenes, it is possible to infer depth information about the points in each image.

A stereo pair is cheap, but perhaps the least accurate sensor. Ideally, it would require perfect calibration of both cameras, which is unfeasible in practice. Bad light conditions will render it useless. Also, because of the way the algorithm works (detecting corresponding points of interest in both images), they give poor results with empty scenes or objects that have plain textures, with few interest points. Some models circumvent this by projecting a grid or texture of light on the scene (active stereo vision).

I will not go into detail about the math involved with the triangulation process, as you can find it on the internet.

[[Image:Stereo.png|thumb|center|200px|Example of stereo camera.]]

==Time-of-flight==

[http://en.wikipedia.org/wiki/Time_of_flight Time-of-flight] (ToF) sensors work by measuring the time it has taken a ray or pulse of light to travel a certain distance. Because the speed of light is a known constant, a simple formula can be used to obtain the range to the object. These sensors are not affected by light conditions and have the potential to be very precise.

===LIDAR===

A [http://en.wikipedia.org/wiki/Lidar LIDAR] (light+radar) is just a common laser range finder mounted on a platform that is able to rotate very fast, scanning the scene point by point. They are very precise sensors, but also expensive, and they do not retrieve texture information. They have been used for decades in many different fields like meteorology, archaeology or astronomy. LIDAR devices can be mounted on satellites, planes or mobile robots. The data retrieved by a LIDAR has very high resolution, so some processing is needed in order to use it for real-time applications.

[[Image:LIDAR.jpg|thumb|center|200px|LIDAR mounted on a mobile robot (image from Wikimedia Commons).]]

===ToF cameras===

A [http://en.wikipedia.org/wiki/Time-of-flight_camera time-of-flight camera] does not perform point-by-point scans like a LIDAR does. Instead, it employs a single pulse of light to capture the whole scene, once per frame. Thanks to that, they can work a lot faster, with some models topping above 100 Hz. The price to pay is a low resolution of 320×240 or even less. Depth measurements have an accuracy of about 1cm. ToF cameras cost less than LIDAR sensors, but we are talking about 4000 $ or so. Color information is not retrieved.

The new [https://dev.windows.com/en-us/kinect Kinect v2] is a ToF sensor that works at 512×424 internally, and it includes a 1920x1080 RGB camera.

<center><gallery widths=300px>
File:ToFcam.png | Time-of-flight camera (image from Wikimedia Commons).
File:Kinect2.png | Microsoft Kinect v2.
</gallery></center>

==Structured light==

[http://en.wikipedia.org/wiki/Structured-light_3D_scanner Structured light] sensors (like Kinect and Xtion) work by projecting a pattern of infrarred light (for example, a grid of lines, or a "constellation" of points) on top of the scene's objects. This pattern is seen distorted when looked from a perspective different from the projector's. By analysing this distortion, information about the depth can be retrieved, and the surface(s) reconstructed.

The resolution and speed of these sensors are similar to common VGA cameras, usually 640x480 at 30 FPS (actually less because of the way this type of sensors work, Kinect uses 320x240 internally, the rest of the data is extrapolated to "fill the gaps"). Precision is similar to that of ToF cameras, 1cm more or less, with a maximum range of 3-6m, but they tend to have trouble seeing small objects. They are a lot cheaper than any other sensors, with a first generation Kinect now costing around 100 $, and hence they have become a lot popular in the last years. In the next tutorial, I will explain the installation and usage of one of these cameras.

<center><gallery widths=300px>
File:Kinect.jpg | Microsoft Kinect.
File:Xtion.jpg | ASUS Xtion PRO (the PRO LIVE model includes a RGB camera).
</gallery></center>

[[Image:Structured_light.png|thumb|center|200px|Infrarred light pattern (dots) projected by Kinect.]]

=Point clouds=

[[Image:Point_cloud.png|thumb|right|200px|Example of a point cloud.]]

Depth sensors return 3D data in the form of a [http://en.wikipedia.org/wiki/Point_cloud ''point cloud'']. A point cloud is a set of points in three-dimensional space, each with its own XYZ coordinates. In the case of stereo, ToF or structured light cameras, every point corresponds to exactly one pixel of the captured image. Optionally, points can store additional information, like color (if the sensor has a RGB camera that it uses to put texture on them). If you have background with 3D modelling software, then you must know that point clouds can be converted to triangular meshes with certain algorithms, but this is rarely done, and cloud processing is performed with the original set of data.

==Point Cloud Library==

[http://www.pointclouds.org/ Point Cloud Library (PCL)] is a project started in early 2010 by [http://www.willowgarage.com/ Willow Garage], the same robotics research company behind the [http://www.ros.org/ Robot Operating System (ROS)] and OpenCV (and they also sell [http://www.willowgarage.com/robot/overview robots] like TurtleBot, too). The first version was [http://www.pointclouds.org/assets/pdf/pcl_icra2011.pdf fully introduced] in 2011, and it has been actively maintained ever since.

PCL aims to be an one-for-all solution for point cloud and 3D processing. It is an open source, multiplatform library divided in many [http://pointclouds.org/documentation/tutorials/walkthrough.php submodules] for different tasks, like visualization, filtering, segmentation, registration, searching, feature estimation... Additionally, it can be used to retrieve data from a list of compatible sensors, so it is an obvious choice for our tutorials, because you will not have to compile or install dozens of separate libraries.

Apart from Willow Garage, PCL is backed by [http://www.openperception.org/ Open Perception], a non-profit organization founded by Radu Bogdan Rusu, one of the most active developers of the library. I recommend you to read his very interesting [http://files.rbrusu.com/publications/RusuPhDThesis.pdf dissertation], written in 2009, which introduces and details many concepts and techniques that have been implemented into PCL.

=Conclusion=

Over the course of the next tutorials I will tell you how to install PCL in a Linux environment, how to retrieve point clouds from a sensor like Kinect or Xtion, and how to work with them for things like object recognition. I will keep it nice and easy, just like I want tutorials to be, and leave complex stuff aside (you can always search the original paper if you want to learn more about some technique).

----
----

Go to root: [[PhD-3D-Object-Tracking]]

Links to articles:

[[PCL/OpenNI tutorial 0: The very basics]]

[[PCL/OpenNI tutorial 1: Installing and testing]]

[[PCL/OpenNI tutorial 2: Cloud processing (basic)]]

[[PCL/OpenNI tutorial 3: Cloud processing (advanced)]]

[[PCL/OpenNI tutorial 4: 3D object recognition (descriptors)]]

[[PCL/OpenNI tutorial 5: 3D object recognition (pipeline)]]

[[PCL/OpenNI troubleshooting]]