Difference between revisions of "DMartinez-PFC-Nao"
m |
|||
| (38 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | * ''' | + | * '''Author:''' David Martínez Martínez (davidmm1988 [at] gmail [dot] com) |
| + | * '''Academic Year:''' 2009-2010 | ||
| + | * '''Degree:''' Undergraduate | ||
| + | * '''Tags:''' webots, naoqi, jmanager | ||
| + | * '''Technologies:''' java, c++, jmf | ||
| + | * '''Status:''' Finished | ||
| + | * '''Source License:''' [http://www.gnu.org/licenses/gpl-3.0-standalone.html GPLv3] | ||
| + | |||
| + | |||
| + | =Teleoperator= | ||
| + | |||
| + | ==Features== | ||
| + | |||
| + | ===Teleoperate Nao=== | ||
| + | |||
| + | ===Displaying Nao's camera and joints' positions=== | ||
| + | |||
| + | The teleoperator can display what Nao can see with its camera, and the current position of its joints. | ||
| + | |||
| + | It creates a thread for video and another for the joints which will tell him to update the image/joints continuously. | ||
| − | + | It uses the existing connection with the robot to ask for the images and the joints. | |
| − | + | Diagrama de clases involucradas: | |
| + | [[Image:Teleoperator diag clases get display info.png|center|950px]] | ||
| + | |||
| + | Diagrama de secuencia del proceso: | ||
| + | [[Image:Teleoperator secuencia get display info.png|center|730px]] | ||
| + | |||
| + | ===Recording Nao's camera and joints' positions=== | ||
| + | |||
| + | You can record the video and joints positions which jmanager receives. It stores the video as raw RGB in a avi container with a fixed framerate. The joints are saved in a binary file that can be opened with the jmanager. Joints are also saved in a SRT subtitle so they can be played in any video player which supports them. | ||
| + | |||
| + | The teleoperator can play recorded videos and joints files. They will be displayed in its interface as if they were taken from the robot. | ||
| + | |||
| + | It uses JMF to process, mux and demux video. | ||
| + | |||
| + | The class FrameSynchronizer implements the required methods to synchronize the video and joints recording threads at a fixed frameRate. A abstract class exists for video files processing, and also another one for joints files processing, which makes creating new recorders and player very easy. | ||
| + | |||
| + | |||
| + | Diagrama de clases involucradas: | ||
| + | |||
| + | [[Image:Teleoperator diag clases display and record.png|center|770px]] | ||
| − | |||
| − | + | Diagrama de secuencia del proceso del teleoperador para grabar video a un archivo: | |
| − | + | [[Image:Teleoperator secuencia recorder video.png|center|914px]] | |
| − | |||
| − | + | Diagrama de secuencia del proceso del teleoperador para grabar las posiciones de las joints a un archivo: | |
| − | + | [[Image:Teleoperator secuencia recorder joints.png|center|811px]] | |
| − | |||
| − | + | Diagrama de secuencia del proceso del teleoperador para reproducir un archivo de video en su interfaz: | |
| − | + | [[Image:Teleoperator secuencia player video.png|center|813px]] | |
| + | Diagrama de secuencia del proceso del teleoperador para reproducir un archivo de posiciones de articulaciones: | ||
| + | [[Image:Teleoperator secuencia player joints.png|center|833px]] | ||
==Versions== | ==Versions== | ||
===0.1 Teleoperator=== | ===0.1 Teleoperator=== | ||
| + | |||
Status: Completed | Status: Completed | ||
| + | A basic teleoperator to move Nao. It can use BICA's body or a new component to do the movements. | ||
| + | |||
| + | [[Image:teleoperator_version_0.1-r1.png|center|844px]] | ||
===0.2 Video & Joints display=== | ===0.2 Video & Joints display=== | ||
| + | |||
Status: Completed | Status: Completed | ||
| + | The teleoperator can show the video captured from Nao's camera, and the position of its joints. | ||
| + | |||
| + | [[Image:teleoperator_version_0.2-r1.png|center|950px]] | ||
===0.3 Video & Joints recorder=== | ===0.3 Video & Joints recorder=== | ||
| + | |||
Status: Completed | Status: Completed | ||
| + | Added the posibility to save the video and joints states to a file. | ||
| + | |||
| + | [[Image:teleoperator_version_0.3.png|center|950px]] | ||
===0.4 Video & Joints player=== | ===0.4 Video & Joints player=== | ||
| + | |||
Status: Completed | Status: Completed | ||
| + | The teleoperator can open a play a recorded video/joints file on its interface. | ||
| + | |||
| + | [[Image:teleoperator_version_0.4.png|center|950px]] | ||
===0.5 Migrating to naoqi 1.6=== | ===0.5 Migrating to naoqi 1.6=== | ||
| − | |||
| + | Status: Completed. | ||
| + | |||
| + | Code modified to use the last version of naoqi (1.6). The movement is now unified. | ||
| + | Now only BICA's movement is supported. | ||
===0.6 Interface improved=== | ===0.6 Interface improved=== | ||
| + | |||
| + | Status: Completed. | ||
| + | |||
| + | Making the interface more appealing. | ||
| + | |||
| + | [[Image:teleoperator_version_0.6.png|center|950px]] | ||
| + | |||
| + | |||
| + | ===1.0 Final=== | ||
| + | |||
Status: Development. | Status: Development. | ||
| + | Bug fixing. | ||
| + | |||
| + | <screenshot> | ||
==Journal== | ==Journal== | ||
| − | + | ; 3 November 2009 | |
| − | |||
Primeras pruebas de programación para el Nao. | Primeras pruebas de programación para el Nao. | ||
| Line 63: | Line 131: | ||
| − | + | ; 4 November 2009 | |
Comienzo el segundo tutorial, pero lo dejo a medias debido a un problema en el make al crear un proxy sin broker. | Comienzo el segundo tutorial, pero lo dejo a medias debido a un problema en el make al crear un proxy sin broker. | ||
| − | + | ; 5 November 2009 | |
Conseguí realizar el segundo tutorial, haciendo los cambios que me indicó Juan para conseguir compilar el proxy sin broker. El resto los conseguí compilar bien adecuando el código de ejemplo a los archivos generados por el moduleGenerator. | Conseguí realizar el segundo tutorial, haciendo los cambios que me indicó Juan para conseguir compilar el proxy sin broker. El resto los conseguí compilar bien adecuando el código de ejemplo a los archivos generados por el moduleGenerator. | ||
| − | + | ;18 November 2009 | |
He estado probando el webots, ejecutando los brokers sobre los nao que salen en el webots. | He estado probando el webots, ejecutando los brokers sobre los nao que salen en el webots. | ||
También instalé la nueva versión del naoqi, que funciona bien en la máquina virtual del ubuntu 9.04. | También instalé la nueva versión del naoqi, que funciona bien en la máquina virtual del ubuntu 9.04. | ||
| Line 79: | Line 147: | ||
| − | + | ; 9 January 2010 | |
Hice las primeras pruebas con el manager, las cuales no fueron muy productivas. | Hice las primeras pruebas con el manager, las cuales no fueron muy productivas. | ||
| Line 86: | Line 154: | ||
| − | + | ; 10 January 2010 | |
Solventé los problemas que había tenido con el manager. | Solventé los problemas que había tenido con el manager. | ||
| Line 95: | Line 163: | ||
| − | + | ; 8 February 2010 | |
Comienzo a trastear con el manager. | Comienzo a trastear con el manager. | ||
| Line 102: | Line 170: | ||
| − | + | ; 9 February 2010 | |
Continuo con los primeros pasos en el manager. | Continuo con los primeros pasos en el manager. | ||
| Line 109: | Line 177: | ||
| − | + | ; 19 February 2010 | |
Fui al laboratorio, donde Fran me estuvo enseñando en nuevo jmanager y el player. | Fui al laboratorio, donde Fran me estuvo enseñando en nuevo jmanager y el player. | ||
| Line 115: | Line 183: | ||
| − | + | ; 26 February 2010 | |
Ya tengo la versión final del jmanager. Después de diversos problemas con gluegen y jogl consigo ejecutarlo desde consola, aunque sigo sin conseguir ejecutarlo desde el netbeans. | Ya tengo la versión final del jmanager. Después de diversos problemas con gluegen y jogl consigo ejecutarlo desde consola, aunque sigo sin conseguir ejecutarlo desde el netbeans. | ||
| Line 121: | Line 189: | ||
| − | + | ; 28 February 2010 | |
Hecha una primera implementación del teleoperador. | Hecha una primera implementación del teleoperador. | ||
Muestra el video, y utiliza el componente Body para mover al Nao. Responde mal a los movimientos. | Muestra el video, y utiliza el componente Body para mover al Nao. Responde mal a los movimientos. | ||
| − | + | ; 1 March 2010 | |
Hecha una segunda implementación. Utiliza un componente propio que hace llamadas a Almotion, pero que se salta las guías de BICA (no utiliza step() ni isTimeToRun() ). Funciona bastante bien. | Hecha una segunda implementación. Utiliza un componente propio que hace llamadas a Almotion, pero que se salta las guías de BICA (no utiliza step() ni isTimeToRun() ). Funciona bastante bien. | ||
| − | + | ; 1 March 2010 | |
I start using subversion. | I start using subversion. | ||
| − | + | ; 5 March 2010 | |
Now the operator only moves Nao while the buttons are being pressed. Before Nao moved until stop key was pressed. | Now the operator only moves Nao while the buttons are being pressed. Before Nao moved until stop key was pressed. | ||
| − | + | ; 8 March 2010 | |
Now the teleoperator can also move the Head. It uses BICA code on Nao's side, and it processes key events on the jmanager's side. | Now the teleoperator can also move the Head. It uses BICA code on Nao's side, and it processes key events on the jmanager's side. | ||
| − | + | ; 15 March 2010 | |
I start doing the code for modifying joints positions. It uses a new controller in nao's player which is already completed. Only a few basic functions are implemented on Jmanager yet. | I start doing the code for modifying joints positions. It uses a new controller in nao's player which is already completed. Only a few basic functions are implemented on Jmanager yet. | ||
| − | + | ; 17 March 2010 | |
The hard drive died, maybe because a firmware problem, and the last changes for modifying joints weren't uploaded. Maybe I'll be able to recover that when it's repaired. | The hard drive died, maybe because a firmware problem, and the last changes for modifying joints weren't uploaded. Maybe I'll be able to recover that when it's repaired. | ||
| − | + | ; 1 April 2010 | |
First steps with JMF. JMF has many problems, but I can't find any alternatives. | First steps with JMF. JMF has many problems, but I can't find any alternatives. | ||
I couldn't find a updated wrapper for FFMPEG to use libav*'s API, the latest was from 2008. Gstreamer doesn't support manually adding or removing data from/to a pipeline, which is required to get/set the data to the teleoperator. | I couldn't find a updated wrapper for FFMPEG to use libav*'s API, the latest was from 2008. Gstreamer doesn't support manually adding or removing data from/to a pipeline, which is required to get/set the data to the teleoperator. | ||
| − | + | ; 3 April 2010 | |
JMF working. The teleoperator can save the video to a file. | JMF working. The teleoperator can save the video to a file. | ||
| − | + | ; 5 April 2010 | |
| − | Reestructured the code for saving video. Now it records video at a given framerate. | + | Reestructured the code for saving video. Now it records video at a given framerate. Made synchronization methods. |
| + | |||
| + | |||
| + | ; 6 April 2010 | ||
| + | Made a sensor class for getting joints states from Nao. | ||
| + | Teleoperator gets and shows joints states from Nao. | ||
| + | |||
| + | |||
| + | ; 7 April 2010 | ||
| + | Now it records joints states. | ||
| + | |||
| + | |||
| + | ; 8 April 2010 | ||
| + | Synchronization code splitted for having more readable code. Synchronization fixes. | ||
| + | |||
| + | |||
| + | ; 9 April 2010 | ||
| + | Teleoperator can now play recorded video. | ||
| + | Recording threads synchronization finally works without any visible isues. | ||
| + | |||
| + | |||
| + | ; 10 April 2010 | ||
| + | Now a subtitle with joints states is also made. | ||
| + | |||
| + | |||
| + | ; 19 April 2010 | ||
| + | This week I have been doing documentation and updating the wiki. Also some minor bugs have been fixed. | ||
| + | |||
| + | |||
| + | ; 26 April 2010 | ||
| + | Migrating to naoqi 1.6. Found some issues with the movement and capturing images. | ||
| + | Fixed the problems when jmanager wouldn't respond due to a packet loss. Now it asked Nao for data in a ordered secuence. | ||
| − | |||
| + | ; 4 May 2010 | ||
| + | Improved the interface which was very ugly. | ||
| + | Added the option to set the fps of recorded and played video. | ||
| − | |||
| − | |||
| − | + | ; 14 May 2010 | |
| + | Changed the internal arquitecture to be more modular and readable. Tested with the real robot. Everything seems to work. | ||
| − | + | ; 16 May 2010 | |
| − | + | Documented the technical part of the application. | |
| − | + | ; 20 May 2010 | |
| − | + | First version of the documentation almost finished. | |
Latest revision as of 08:37, 25 June 2014
- Author: David Martínez Martínez (davidmm1988 [at] gmail [dot] com)
- Academic Year: 2009-2010
- Degree: Undergraduate
- Tags: webots, naoqi, jmanager
- Technologies: java, c++, jmf
- Status: Finished
- Source License: GPLv3
Teleoperator
Features
Teleoperate Nao
Displaying Nao's camera and joints' positions
The teleoperator can display what Nao can see with its camera, and the current position of its joints.
It creates a thread for video and another for the joints which will tell him to update the image/joints continuously.
It uses the existing connection with the robot to ask for the images and the joints.
Diagrama de clases involucradas:
Diagrama de secuencia del proceso:
Recording Nao's camera and joints' positions
You can record the video and joints positions which jmanager receives. It stores the video as raw RGB in a avi container with a fixed framerate. The joints are saved in a binary file that can be opened with the jmanager. Joints are also saved in a SRT subtitle so they can be played in any video player which supports them.
The teleoperator can play recorded videos and joints files. They will be displayed in its interface as if they were taken from the robot.
It uses JMF to process, mux and demux video.
The class FrameSynchronizer implements the required methods to synchronize the video and joints recording threads at a fixed frameRate. A abstract class exists for video files processing, and also another one for joints files processing, which makes creating new recorders and player very easy.
Diagrama de clases involucradas:
Diagrama de secuencia del proceso del teleoperador para grabar video a un archivo:
Diagrama de secuencia del proceso del teleoperador para grabar las posiciones de las joints a un archivo:
Diagrama de secuencia del proceso del teleoperador para reproducir un archivo de video en su interfaz:
Diagrama de secuencia del proceso del teleoperador para reproducir un archivo de posiciones de articulaciones:
Versions
0.1 Teleoperator
Status: Completed
A basic teleoperator to move Nao. It can use BICA's body or a new component to do the movements.
0.2 Video & Joints display
Status: Completed
The teleoperator can show the video captured from Nao's camera, and the position of its joints.
0.3 Video & Joints recorder
Status: Completed
Added the posibility to save the video and joints states to a file.
0.4 Video & Joints player
Status: Completed
The teleoperator can open a play a recorded video/joints file on its interface.
0.5 Migrating to naoqi 1.6
Status: Completed.
Code modified to use the last version of naoqi (1.6). The movement is now unified. Now only BICA's movement is supported.
0.6 Interface improved
Status: Completed.
Making the interface more appealing.
1.0 Final
Status: Development.
Bug fixing.
<screenshot>
Journal
- 3 November 2009
Primeras pruebas de programación para el Nao.
- Como el sistema operativo que tengo es de 64bits y las herramientas de programación del Nao vienen preparadas para 32 bits, creo una máquina virtual de ubuntu 9.04 de 32 bits sobre VirtualBox para programar en ella.
- Pruebo las herramientas y realizo el primer tutorial de http://www.teamchaos.es/index.php/Nao para ver que todo funciona.
- 4 November 2009
Comienzo el segundo tutorial, pero lo dejo a medias debido a un problema en el make al crear un proxy sin broker.
- 5 November 2009
Conseguí realizar el segundo tutorial, haciendo los cambios que me indicó Juan para conseguir compilar el proxy sin broker. El resto los conseguí compilar bien adecuando el código de ejemplo a los archivos generados por el moduleGenerator.
- 18 November 2009
He estado probando el webots, ejecutando los brokers sobre los nao que salen en el webots. También instalé la nueva versión del naoqi, que funciona bien en la máquina virtual del ubuntu 9.04.
Probé a crear una máquina virtual del ubuntu 9.10, pero daba problemas con virtualbox. Tendré que probar a actualizar el virtualbox a la última versión. También probé a utilizar naoqi y webots desde el SO principal que uso, pero al ser de 64 bits daba problemas, ya que no tengo las librerías de python de 32 bits.
- 9 January 2010
Hice las primeras pruebas con el manager, las cuales no fueron muy productivas.
Compilé el player y el manager, y conecté el manager al nao del webots. No conseguí activar la cámara, ya que me devolvía algunos errores sobre archivos de configuración. La máquina virtual tampoco respondía muy bien.
- 10 January 2010
Solventé los problemas que había tenido con el manager.
Por un lado no tenía los archivos de configuración de /var/lib completos. Por otro no estaba activando la opción de debug que era necesaria para que la conexión con el Nao funcionase bien.
Con esto ya funciona todo bastante bien.
- 8 February 2010
Comienzo a trastear con el manager.
Creé una pestaña nueva donde empezar a construir un teleoperador. Basándome en el código ya hecho para la pestaña de la cámara, introduje en mi pestaña soporte para ver el video de lo que está viendo el robot.
- 9 February 2010
Continuo con los primeros pasos en el manager.
Esta vez introduje un botón para mover hacia adelante al Nao. Para ello también tuve que modificar el código del player. Introduje en coach.cpp un nuevo método para activar el movimiento hacia adelante, fijándome para ello en la interfaz de los métodos ya existentes.
- 19 February 2010
Fui al laboratorio, donde Fran me estuvo enseñando en nuevo jmanager y el player.
Empecé a trastear con el jmanager, creando una nueva pestaña en la que poner la cámara del robot y un botón para iniciar el movimiento.
- 26 February 2010
Ya tengo la versión final del jmanager. Después de diversos problemas con gluegen y jogl consigo ejecutarlo desde consola, aunque sigo sin conseguir ejecutarlo desde el netbeans.
He montado un chroot de 32 bits para compilar los brokers desde mi máquina sin necesidad de una máquina virtual. El webots y el naoqi se pueden ejecutar perfectamente en un entorno de 64 bits.
- 28 February 2010
Hecha una primera implementación del teleoperador. Muestra el video, y utiliza el componente Body para mover al Nao. Responde mal a los movimientos.
- 1 March 2010
Hecha una segunda implementación. Utiliza un componente propio que hace llamadas a Almotion, pero que se salta las guías de BICA (no utiliza step() ni isTimeToRun() ). Funciona bastante bien.
- 1 March 2010
I start using subversion.
- 5 March 2010
Now the operator only moves Nao while the buttons are being pressed. Before Nao moved until stop key was pressed.
- 8 March 2010
Now the teleoperator can also move the Head. It uses BICA code on Nao's side, and it processes key events on the jmanager's side.
- 15 March 2010
I start doing the code for modifying joints positions. It uses a new controller in nao's player which is already completed. Only a few basic functions are implemented on Jmanager yet.
- 17 March 2010
The hard drive died, maybe because a firmware problem, and the last changes for modifying joints weren't uploaded. Maybe I'll be able to recover that when it's repaired.
- 1 April 2010
First steps with JMF. JMF has many problems, but I can't find any alternatives. I couldn't find a updated wrapper for FFMPEG to use libav*'s API, the latest was from 2008. Gstreamer doesn't support manually adding or removing data from/to a pipeline, which is required to get/set the data to the teleoperator.
- 3 April 2010
JMF working. The teleoperator can save the video to a file.
- 5 April 2010
Reestructured the code for saving video. Now it records video at a given framerate. Made synchronization methods.
- 6 April 2010
Made a sensor class for getting joints states from Nao. Teleoperator gets and shows joints states from Nao.
- 7 April 2010
Now it records joints states.
- 8 April 2010
Synchronization code splitted for having more readable code. Synchronization fixes.
- 9 April 2010
Teleoperator can now play recorded video. Recording threads synchronization finally works without any visible isues.
- 10 April 2010
Now a subtitle with joints states is also made.
- 19 April 2010
This week I have been doing documentation and updating the wiki. Also some minor bugs have been fixed.
- 26 April 2010
Migrating to naoqi 1.6. Found some issues with the movement and capturing images. Fixed the problems when jmanager wouldn't respond due to a packet loss. Now it asked Nao for data in a ordered secuence.
- 4 May 2010
Improved the interface which was very ugly. Added the option to set the fps of recorded and played video.
- 14 May 2010
Changed the internal arquitecture to be more modular and readable. Tested with the real robot. Everything seems to work.
- 16 May 2010
Documented the technical part of the application.
- 20 May 2010
First version of the documentation almost finished.
