ROS kinetic 机器视觉

ROS 中的图像数据
二维图像
显示图像类型
[email protected]:~$ roslaunch usb_cam usb_cam-test.launch
[email protected]:~$ rostopic list
/image_view/output
/image_view/parameter_descriptions
/image_view/parameter_updates
/rosout
/rosout_agg
/usb_cam/camera_info
/usb_cam/image_raw
/usb_cam/image_raw/compressed
/usb_cam/image_raw/compressed/parameter_descriptions
/usb_cam/image_raw/compressed/parameter_updates
/usb_cam/image_raw/compressedDepth
/usb_cam/image_raw/compressedDepth/parameter_descriptions
/usb_cam/image_raw/compressedDepth/parameter_updates
/usb_cam/image_raw/theora
/usb_cam/image_raw/theora/parameter_descriptions
/usb_cam/image_raw/theora/parameter_updates
[email protected]:~$ rostopic info /usb_cam/image_raw
Type: sensor_msgs/Image

Publishers:
* /usb_cam (http://vm:34615/)

Subscribers:
* /image_view (http://vm:38655/)

[email protected]:~$ rosmsg show sensor_msgs/Image
std_msgs/Header header //消息头，包含消息序号，时间戳和绑定坐标关系；
        uint32 seq // 消息的序号
        time stamp // 时间戳
        string frame_id // 绑定坐标关系
uint32 height // 图像的纵向分辨率；
uint32 width // 图像的横向分辨率；
string encoding // 图像的编码格式，包含RGB 、YUV等常用格式，不涉及图像压缩编码；
uint8 is_bigendian // 图像数据的大小端存储模式；
uint32 step //一行图像数据的字节数量，作为数据的步长参数；
uint8[] data // 存储图像数据的数组，大小为step * heigh 个字节;

如：1080*720分辨率的摄像头产生一帧图像的数据大小是3*1080*720

查看压缩图像
[email protected]:~$ rqt_image_view
[email protected]:~$ rostopic info /usb_cam/image_raw/compressed
Type: sensor_msgs/CompressedImage
Publishers:
* /usb_cam (http://T430:40229/)
Subscribers:
* /rqt_gui_cpp_node_5882 (http://T430:37783/)

图像压缩消息
[email protected]:~$ rosmsg show sensor_msgs/CompressedImage
std_msgs/Header header
        uint32 seq
        time stamp
        string frame_id
string format      // 图像压缩编码格式(jpeg、png、bmp)

uint8[] data // 存储图像数据的数组

三维图像
显示点云类型
[email protected]:~$ roslaunch freenect_launch freenect.launch
[email protected]:~$ rostopic list
/camera/depth/camera_info
/camera/depth/disparity
/camera/depth/image
/camera/depth/image/compressed
/camera/depth/image/compressed/parameter_descriptions
/camera/depth/image/compressed/parameter_updates
/camera/depth/image/compressedDepth
/camera/depth/image/compressedDepth/parameter_descriptions
/camera/depth/image/compressedDepth/parameter_updates
/camera/depth/image/theora
/camera/depth/image/theora/parameter_descriptions
/camera/depth/image/theora/parameter_updates
/camera/depth/image_raw
/camera/depth/image_raw/compressed
/camera/depth/image_raw/compressed/parameter_descriptions
/camera/depth/image_raw/compressed/parameter_updates
/camera/depth/image_raw/compressedDepth
/camera/depth/image_raw/compressedDepth/parameter_descriptions
/camera/depth/image_raw/compressedDepth/parameter_updates
/camera/depth/image_raw/theora
/camera/depth/image_raw/theora/parameter_descriptions
/camera/depth/image_raw/theora/parameter_updates
/camera/depth/image_rect
/camera/depth/image_rect/compressed
/camera/depth/image_rect/compressed/parameter_descriptions
/camera/depth/image_rect/compressed/parameter_updates
/camera/depth/image_rect/compressedDepth
/camera/depth/image_rect/compressedDepth/parameter_descriptions
/camera/depth/image_rect/compressedDepth/parameter_updates
/camera/depth/image_rect/theora
/camera/depth/image_rect/theora/parameter_descriptions
/camera/depth/image_rect/theora/parameter_updates
/camera/depth/image_rect_raw
/camera/depth/image_rect_raw/compressed
/camera/depth/image_rect_raw/compressed/parameter_descriptions
/camera/depth/image_rect_raw/compressed/parameter_updates
/camera/depth/image_rect_raw/compressedDepth
/camera/depth/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth/image_rect_raw/compressedDepth/parameter_updates
/camera/depth/image_rect_raw/theora
/camera/depth/image_rect_raw/theora/parameter_descriptions
/camera/depth/image_rect_raw/theora/parameter_updates
/camera/depth/points
/camera/depth_rectify_depth/parameter_descriptions
/camera/depth_rectify_depth/parameter_updates
/camera/depth_registered/camera_info
/camera/depth_registered/disparity
/camera/depth_registered/hw_registered/image_rect
/camera/depth_registered/hw_registered/image_rect/compressed
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect/compressedDepth
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect/theora
/camera/depth_registered/hw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw
/camera/depth_registered/hw_registered/image_rect_raw/compressed
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/theora
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered/image
/camera/depth_registered/image/compressed
/camera/depth_registered/image/compressed/parameter_descriptions
/camera/depth_registered/image/compressed/parameter_updates
/camera/depth_registered/image/compressedDepth
/camera/depth_registered/image/compressedDepth/parameter_descriptions
/camera/depth_registered/image/compressedDepth/parameter_updates
/camera/depth_registered/image/theora
/camera/depth_registered/image/theora/parameter_descriptions
/camera/depth_registered/image/theora/parameter_updates
/camera/depth_registered/image_raw
/camera/depth_registered/image_raw/compressed
/camera/depth_registered/image_raw/compressed/parameter_descriptions
/camera/depth_registered/image_raw/compressed/parameter_updates
/camera/depth_registered/image_raw/compressedDepth
/camera/depth_registered/image_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/image_raw/compressedDepth/parameter_updates
/camera/depth_registered/image_raw/theora
/camera/depth_registered/image_raw/theora/parameter_descriptions
/camera/depth_registered/image_raw/theora/parameter_updates
/camera/depth_registered/points
/camera/depth_registered/sw_registered/camera_info
/camera/depth_registered/sw_registered/image_rect
/camera/depth_registered/sw_registered/image_rect/compressed
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect/compressedDepth
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect/theora
/camera/depth_registered/sw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw
/camera/depth_registered/sw_registered/image_rect_raw/compressed
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/theora
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered_rectify_depth/parameter_descriptions
/camera/depth_registered_rectify_depth/parameter_updates
/camera/driver/parameter_descriptions
/camera/driver/parameter_updates
/camera/ir/camera_info
/camera/ir/image_raw
/camera/ir/image_raw/compressed
/camera/ir/image_raw/compressed/parameter_descriptions
/camera/ir/image_raw/compressed/parameter_updates
/camera/ir/image_raw/compressedDepth
/camera/ir/image_raw/compressedDepth/parameter_descriptions
/camera/ir/image_raw/compressedDepth/parameter_updates
/camera/ir/image_raw/theora
/camera/ir/image_raw/theora/parameter_descriptions
/camera/ir/image_raw/theora/parameter_updates
/camera/ir/image_rect_ir
/camera/ir/image_rect_ir/compressed
/camera/ir/image_rect_ir/compressed/parameter_descriptions
/camera/ir/image_rect_ir/compressed/parameter_updates
/camera/ir/image_rect_ir/compressedDepth
/camera/ir/image_rect_ir/compressedDepth/parameter_descriptions
/camera/ir/image_rect_ir/compressedDepth/parameter_updates
/camera/ir/image_rect_ir/theora
/camera/ir/image_rect_ir/theora/parameter_descriptions
/camera/ir/image_rect_ir/theora/parameter_updates
/camera/ir_rectify_ir/parameter_descriptions
/camera/ir_rectify_ir/parameter_updates
/camera/projector/camera_info
/camera/rgb/camera_info
/camera/rgb/image_color
/camera/rgb/image_color/compressed
/camera/rgb/image_color/compressed/parameter_descriptions
/camera/rgb/image_color/compressed/parameter_updates
/camera/rgb/image_color/compressedDepth
/camera/rgb/image_color/compressedDepth/parameter_descriptions
/camera/rgb/image_color/compressedDepth/parameter_updates
/camera/rgb/image_color/theora
/camera/rgb/image_color/theora/parameter_descriptions
/camera/rgb/image_color/theora/parameter_updates
/camera/rgb/image_mono
/camera/rgb/image_mono/compressed
/camera/rgb/image_mono/compressed/parameter_descriptions
/camera/rgb/image_mono/compressed/parameter_updates
/camera/rgb/image_mono/compressedDepth
/camera/rgb/image_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_mono/compressedDepth/parameter_updates
/camera/rgb/image_mono/theora
/camera/rgb/image_mono/theora/parameter_descriptions
/camera/rgb/image_mono/theora/parameter_updates
/camera/rgb/image_raw
/camera/rgb/image_raw/compressed
/camera/rgb/image_raw/compressed/parameter_descriptions
/camera/rgb/image_raw/compressed/parameter_updates
/camera/rgb/image_raw/compressedDepth
/camera/rgb/image_raw/compressedDepth/parameter_descriptions
/camera/rgb/image_raw/compressedDepth/parameter_updates
/camera/rgb/image_raw/theora
/camera/rgb/image_raw/theora/parameter_descriptions
/camera/rgb/image_raw/theora/parameter_updates
/camera/rgb/image_rect_color
/camera/rgb/image_rect_color/compressed
/camera/rgb/image_rect_color/compressed/parameter_descriptions
/camera/rgb/image_rect_color/compressed/parameter_updates
/camera/rgb/image_rect_color/compressedDepth
/camera/rgb/image_rect_color/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_color/compressedDepth/parameter_updates
/camera/rgb/image_rect_color/theora
/camera/rgb/image_rect_color/theora/parameter_descriptions
/camera/rgb/image_rect_color/theora/parameter_updates
/camera/rgb/image_rect_mono
/camera/rgb/image_rect_mono/compressed
/camera/rgb/image_rect_mono/compressed/parameter_descriptions
/camera/rgb/image_rect_mono/compressed/parameter_updates
/camera/rgb/image_rect_mono/compressedDepth
/camera/rgb/image_rect_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_mono/compressedDepth/parameter_updates
/camera/rgb/image_rect_mono/theora
/camera/rgb/image_rect_mono/theora/parameter_descriptions
/camera/rgb/image_rect_mono/theora/parameter_updates
/camera/rgb_debayer/parameter_descriptions
/camera/rgb_debayer/parameter_updates
/camera/rgb_rectify_color/parameter_descriptions
/camera/rgb_rectify_color/parameter_updates
/camera/rgb_rectify_mono/parameter_descriptions
/camera/rgb_rectify_mono/parameter_updates
/diagnostics
/rosout
/rosout_agg
/tf
/tf_static
[email protected]:~$
Type: sensor_msgs/PointCloud2
Publishers:
* /camera/camera_nodelet_manager (http://vm:40763/)
Subscribers: None
[email protected]:~$
std_msgs/Header header
        uint32 seq
        time stamp
        string frame_id
uint32 height // 点云图像的纵向分辨率
uint32 width      // 点云图像的横向分辨率
sensor_msgs/PointField[] fields // 每个点的数据类型
        uint8 INT8=1      // 定义了一些常量宏
        uint8 UINT8=2
        uint8 INT16=3
        uint8 UINT16=4
        uint8 INT32=5
        uint8 UINT32=6
        uint8 FLOAT32=7
        uint8 FLOAT64=8
        string name
        uint32 offset
        uint8 datatype
        uint32 count
bool is_bigendian // 数据的大小端存储模式
uint32 point_step // 单点的数据字节步长
uint32 row_step // 一列数据的字节步长
uint8[] data // 点云数据的存储数组，总字节大小为 row_step * height
bool is_dense // 是否有无效点

点云单帧数量很大，如果使用分布式网络传输，需要考虑是否能满足数据的传输要求，或者针对数据进行压缩。

摄像头标定
摄像头这种精密仪器对光学器件的要求较高，摄像头生成的物体图像往往会发生畸变，为避免数据源造成的误差，需要针对摄像头的参数进行标定。
安装标定功能包
[email protected]:~$ sudo apt-get install ros-kinetic-camera-calibration
使用棋盘格标定靶

ROS kinetic 机器视觉

摄像头标定流程
启动摄像头
[email protected]:~$ roslaunch robot_vision usb_cam.launch
启动标定包
[email protected]:~$ rosrun camera_calibration cameracalibrator.py --size 8x6 --square 0.024 image:=/usb_cam/image_raw camera:=/usb_cam

参数说明：
    size ：标定棋盘格的内部角点个数，这里使用的棋盘格一共有6行，每行有8个内部角点；
    square：这个参数对应每个棋盘格的边长，单位是米；一个小正方形的边长为0.024m
    image 和 camera：设置摄像头发布的图像话题

ROS kinetic 机器视觉
X：标定靶在摄像头视野中左右移动；
Y：标定靶在摄像头视野中上下移动；
Size：标定靶在摄像视野中前后移动；
SKew：标定靶在摄像头视野中倾斜转动

都变为绿色，表示满足标定需求，
点击CALIBRATE ，进行计算参数
ROS kinetic 机器视觉
点击SAVE进行保存，路径 ('Wrote calibration data to', '/tmp/calibrationdata.tar.gz') , 解压提取出标定文件 ost.yaml 拷贝到功能包下面。
解压之后，出现标定采集的图像和 ost.txt 、ost.yaml 文件。我们需要使用ost.yaml。
[email protected]:/tmp$ cp ost.yaml ~/catkin_ws/src/robot_vision/camera_calibration.yaml

kinect标定流程
启动Kinect
[email protected]:~$ roslaunch robot_vision freenect.launch
启动彩色摄像头
[email protected]:~$ rosrun camera_calibration cameracalibrator.py image:=/camera/rgb/image_raw camera:=/camera/rgb --size 8x6 --square 0.024
标定红外摄像头
[email protected]:~$ rosrun camera_calibration cameracalibrator.py image:=/camera/ir/image_raw camera:=/camera/ir --size 8x6 --square 0.024

使用标定文件
摄像头

Kinect
<arg name="rgb_camera_info_url" value="file://$(find robot_vision)/kinect_rgb_calibration.yaml" />

使用标定文件可能产生的错误

原因 : 标定文件中的 camera_name 参数与实际传感器名称不匹配
比如根据上图所示的告警，分别将两个标定文件中的 camera_name 参数修改为 rgb_xxxxxxxxxxxx depth_xxxxxxxxxxxxx
image_width: 640
image_height: 480
camera_name: head_camera

camera_matrix:

ROS+OpenCV 应用实例（人脸识别、物体跟踪）
OpenCV ( Open Source Computer Vision Library ) 基于BSD许可发行的跨平台开源计算机视觉库，由一系列C 函数和C++类构成，同时提供C++、Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法，而且对非商业应用和商业应用都是免费的，可以直接访问硬件摄像头，可以直接访问硬件摄像头，并且还提供了一个简单的GUI系统--highgui。
安装OpenCV
[email protected]:~$ sudo apt-get install ros-kinetic-vision-opencv libopencv-dev python-opencv
ROS 与 OpenCV 集成框架

ROS kinetic 机器视觉

运行
[email protected]:~$ roslaunch robot_vision usb_cam.launch
[email protected]:~$ rosrun robot_vision cv_bridge_test.py
[email protected]:~$ rqt_image_view

OpenCV 直接显示：

ROS 通过话题显示：

ROS kinetic 机器视觉

imgmsg_to_cv2()：将ROS图像消息转换成OpenCV图像数据；
cv2_to_imgmsg()：将OpenCV格式的图像数据转换成ROS图像消息；
输入参数：图像消息流和转换的图像数据格式

基于Haar特征的级联分类器对象检测算法

ROS kinetic 机器视觉

初始化
完成ROS节点、图识别参数的设置

ROS 图像回调函数
将图像转换成OpenCV的数据格式，然后预处理之后开始调用人脸识别的功能函数，最后把识别结果发布
    imgmsg_to_cv2(...) 将ROS数据格式的图像转化成OpenCV的图像格式
    cv2.cvtColor(...) 创建灰度图像
    cv2.equalizeHist(...) 创建平衡直方图，减少光线影响

人脸识别
调用OpenCV 提供的人脸识别接口，与数据库中的人脸特征进行匹配

运行
[email protected]:~$ roslaunch robot_vision usb_cam.launch
[email protected]:~$ roslaunch robot_vision face_detector.launch

[email protected]:~$ rqt_image_view

跟踪物体的特征点

ROS kinetic 机器视觉

初始化
完成ROS节点、图识别参数的设置

图像处理
将图像转换成OpenCV格式；完成图像预处理之后开始针对两帧图像进行比较，基于图像差异识别到运行的物体，最后标识识别结果并发布。

运行
[email protected]:~$ roslaunch robot_vision usb_cam.launch
[email protected]:~$ roslaunch robot_vision motion_detector.launch

[email protected]:~$ rqt_image_view

二维码识别

参考网址：wiki.ros.org/ar_track_alvar

修改更新源，阿里云找不到源。
修改为如下源：
deb http://ros.exbot.net/rospackage/ros/ubuntu xenial main

ROS kinetic 机器视觉

ROS kinetic 机器视觉
安装二维码识别功能包
[email protected]:~$ sudo apt-get install ros-kinetic-ar-track-alvar

安装路径
[email protected]:~$ ls /opt/ros/kinetic/share/ar_track_alvar
bundles cmake launch pa

创建二维码
k[email protected]:~$ roscore
[email protected]:~$ rosrun ar_track_alvar createMarker
如：[email protected]:~$ rosrun ar_track_alvar createMarker 0

配置：
[email protected]:~$ roscd robot_vision/config
[email protected]:~/catkin_ws/src/robot_vision/config$ ls
ar_track_camera.rviz ar_track_kinect.rviz

创建大小为5cm的二维码，内容为 0 和 1
[email protected]:~$ rosrun ar_track_alvar createMarker -s 5 0
[email protected]:~$ rosrun ar_track_alvar createMarker -s 5 1
生成的信息保存在 home 目录下

启动摄像头二维码识别示例
摄像头校准文件启动
[email protected]:~$ roslaunch robot_vision usb_cam_with_calibration.launch
[email protected]:~$ roslaunch robot_vision ar_track_camera.launch

启动摄像头，需要加载标定文件，否则可能无法识别二维码。
看到是一个三维的，摄像头达到测距的效果是通过二位大小变化实现的

查看识别到的二维码位姿
[email protected]:~$ rostopic echo /ar_pose_marker

启动kinect二维码识别示例
[email protected]:~$ roslaunch robot_vision freenect.launch
[email protected]:~$ roslaunch robot_vision ar_track_kinect.launch

扩展内容：物体识别与机器学习
Object Recognition Kitchen (ORK)
TensorFlow Object Detection API

参考：
ROS cv_bridge wiki : http://wiki.ros.org/cv_bridge
ROS opencv_apps : http://wiki.ros.org/opencv_apps

ROS kinetic 机器视觉

相关推荐