面向边缘智能光学感知的航空紧固件旋转检测

符长虹; 陈锟辉; 鲁昆瀚; 郑光泽; 赵吉林

doi:10.5768/JAO202243.0303002

面向边缘智能光学感知的航空紧固件旋转检测

doi: 10.5768/JAO202243.0303002

同济大学机械与能源工程学院，上海 201804

基金项目: 上海市“科技创新行动计划”自然科学基金项目（20ZR1460100）

详细信息

作者简介:
符长虹（1986—），男，副教授，工学博士，主要从事智能视觉感知、边缘计算AI视觉系统研制、无人系统自主导航方面的研究。E-mail：changhongfu@tongji.edu.cn

中图分类号: TN925
计量
- 文章访问数: 397
- HTML全文浏览量: 227
- PDF下载量: 36
- 被引次数: 0
出版历程
- 收稿日期: 2022-03-07
- 修回日期: 2022-04-22
- 网络出版日期: 2022-04-24
- 刊出日期: 2022-05-12

Aviation fastener rotation detection for intelligent optical perception with edge computing

School of Mechanical Engineering, Tongji University, Shanghai 201804, China

摘要

摘要: 针对航空紧固件分拣过程中现有方法存在效率低、成本高、精度差等问题，提出一种面向边缘智能光学感知的旋转目标检测方法。构建一种基于强化语义和优化空间的特征融合机制，有效提升目标检测模型的性能；设计一种空洞幻影模块，减少特征融合网络的参数量，有利于模型在工业场景下的边缘部署；采用高斯类环形平滑标签方法，在模型检测层预测分支上实现旋转目标检测，显著提升模型检测性能，并更有助于工业机器人自动抓取。在权威公开旋转数据集上，检测准确率达到77.16%。最后，在嵌入式智能设备上进行边缘部署并测试，整体准确率达到99.76%，推理速度超过20 FPS (frames per second)，满足工业应用的要求。
- 航空紧固件 /
- 旋转目标检测 /
- 特征融合 /
- 空洞幻影模块 /
- 边缘部署
Abstract: Aiming at the problems of low efficiency, high cost and poor accuracy in existing methods in aviation fastener sorting process, a rotation target detection method for intelligent optical perception with edge computing was proposed. To further improve the performance of the target detection model, a feature fusion mechanism based on enhanced semantics and optimized space was constructed. A type of dilated ghost module to lower the parameter quantity of the feature fusion network was designed, and enable the edge computing deployment in industrial scenes. Using the Gaussian-like circular smooth label method, the rotation target detection was realized on the prediction branch of the model detection layer, which significantly enhanced model detection performance and was more favorable for automated grasping of industrial robots. The detection accuracy on the authoritative public rotation dataset reached 77.16%. Finally, the proposed detection method was implemented in an embedded intelligent device. The edge computing deployment shows that the total accuracy reaches 99.76%, and the inference speed is more than 20 frames per second (FPS), which is sufficient for industrial applications.
- aviation fastener /
- rotation target detection /
- feature fusion /
- dilated ghost module /
- edge computing deployment

HTML全文

图 1 轻量级航空紧固件旋转检测方法

Fig. 1 Lightweight aviation fastener rotation detection method

下载: 全尺寸图片幻灯片

图 2 强化语义与优化空间特征融合机制

Fig. 2 Enhanced semantics and optimized space feature fusion mechanism

下载: 全尺寸图片幻灯片

图 3 空洞幻影模块

Fig. 3 Dilated ghost module

下载: 全尺寸图片幻灯片

图 4 瓶颈层

Fig. 4 Bottleneck layer

下载: 全尺寸图片幻灯片

图 5 旋转框角度定义^[19]

Fig. 5 Angle definition of rotation bounding box^[19]

下载: 全尺寸图片幻灯片

图 6 典型航空紧固件

Fig. 6 Typical aviation fasteners

下载: 全尺寸图片幻灯片

图 7 OLAFDet检测结果示例

Fig. 7 Result examples of OLAFDet detection

下载: 全尺寸图片幻灯片

表 1 与前沿旋转框检测方法的对比

Table 1 Comparison with cutting-edge rotation bounding box detection methods

方法	来源	参数量/ MB	PL/%	BD/%	BR/%	GTF/%	SV/%	LV/%	SH/%	TC/%	BC/%	ST/%	SBF/%	RA/%	HA/%	SP/%	HC/%	mAP/%
FR-O^[20]	CVPR2018	242	79.42	77.13	17.70	64.05	35.30	38.02	37.16	89.41	69.64	59.28	50.30	52.91	47.89	47.40	46.30	54.13
TOSO^[21]	ICASSP2020	212	80.17	65.59	39.82	39.95	49.71	65.01	53.58	81.45	44.66	78.51	48.85	56.73	64.40	64.24	36.75	57.92
PIoU Loss^[22]	ECCV2020	−	80.90	69.70	24.10	60.20	38.30	64.40	64.80	90.90	77.20	70.40	46.50	37.10	57.10	61.90	64.00	60.50
Axis Learning^[23]	RS2020	−	79.53	77.15	38.59	61.15	67.53	70.49	76.30	89.66	79.07	83.53	47.27	61.01	56.28	66.06	36.05	65.98
MARNet^[24]	IJRS2021	−	88.91	77.91	39.88	71.17	62.79	58.96	66.25	90.87	73.73	79.04	57.57	64.33	62.47	61.64	51.80	67.15
GSDet^[25]	TIP2021	−	81.12	76.78	40.78	75.89	64.50	58.37	74.21	89.92	79.40	78.83	64.54	63.67	66.04	58.01	52.13	68.28
RADet^[26]	RS2020	−	79.45	76.99	48.05	65.83	65.46	74.40	68.86	89.70	78.14	74.97	49.92	64.63	66.14	71.58	62.16	69.09
RoI Transformer^[27]	CVPR2019	273	88.64	78.52	43.44	75.92	68.81	73.68	83.59	90.74	77.27	81.46	58.39	53.54	62.83	58.93	47.67	69.56
BBAVectors^[28]	WACV2021	276	88.35	79.96	50.69	62.18	78.43	78.98	87.94	90.85	83.58	84.35	54.13	60.24	65.22	64.28	55.70	72.32
SCRDet^[29]	ICCV2019	427	89.98	80.65	52.09	68.36	68.36	60.32	72.41	90.85	87.94	86.86	65.02	66.68	66.25	68.24	65.21	72.61
GLS-Net^[30]	RS2020	−	88.65	77.40	51.20	71.03	73.30	72.16	84.68	90.87	80.43	85.38	58.33	62.27	67.58	70.69	60.42	72.96
R3Det^[6]	AAAI2021	787	88.76	83.09	50.91	67.27	76.23	80.39	86.72	90.78	84.68	83.24	61.98	61.35	66.91	70.63	53.94	73.79
FEDet^[31]	ICCSE2021	−	89.09	79.87	51.13	70.20	78.42	80.54	87.84	90.86	83.91	85.31	58.33	66.90	67.74	69.74	63.43	74.89
TricubeNet^[32]	WACV2022	−	88.75	82.12	49.24	72.98	77.64	74.53	84.65	90.81	86.02	85.38	58.69	63.59	73.82	69.67	71.08	75.26
Beyond Bounding-Box^[33]	CVPR2021	−	89.08	83.20	54.37	66.87	81.23	80.96	87.17	90.21	84.32	86.09	52.34	69.94	75.52	80.76	67.96	76.67
OLAFDet		18	89.28	85.09	48.75	64.65	80.81	84.70	88.09	90.70	86.72	87.41	60.14	67.12	75.02	81.55	67.39	77.16

下载: 导出CSV

表 2 消融分析实验结果

Table 2 Experimental results of ablation analyses

	基准模型			OLAFDet
检测框类型	水平框	旋转框	旋转框	旋转框
高斯类环形平滑标签		√	√	√
强化语义和优化空间			√	√
空洞幻影模块				√
mAP/%	42.32	76.86	77.17	77.16
参数量/MB	14.8	15.8	21.3	18.0

下载: 导出CSV

表 3 航空紧固件检测的测试结果

Table 3 Test results of aviation fastener detection %

类别	螺栓	垫片	螺帽	圆柱销	铆钉	螺钉
AP_s	100.00	100.00	100.00	100.00	100.00	100.00
AP_m	100.00	100.00	98.93	99.99	99.12	100.00
AP	100.00	100.00	99.17	99.99	99.43	100.00
mAP	99.76

下载: 导出CSV

参考文献(33)

[1]	中华人民共和国国民经济和社会发展第十四个五年规划和2035年远景目标纲要[N]. 人民日报, 2021-03-13. Outline of the 14th Five-Year Plan (2021-2025) for national economic and social development and vision 2035 of the People's Republic of China [N]. People's Daily, 2021-03-13. DOI: 10.28655/n.cnki.nrmrb.2021.002455.
[2]	宋瑨, 王世峰. 基于可变形部件模型HOG特征的人形目标检测[J]. 应用光学,2016,37(3):380-384. SONG Jin, WANG Shifeng. Human kind shape object detection using deformable parts model with HOG features[J]. Journal of Applied Optics,2016,37(3):380-384. doi: 10.5768/JAO201637.0302003
[3]	张珹. 高铁接触网支持装置紧固件识别与定位的深度学习方法[J]. 工程数学学报,2020,37(3):261-268. ZHANG Cheng. Deep learning methods for fastener identification and location of high speed railway catenary support devices[J]. Chinese Journal of Engineering Mathematics,2020,37(3):261-268. doi: 10.3969/j.issn.1005-3085.2020.03.001
[4]	王一, 马钲东, 董光林. 基于改进Faster RCNN的零件识别方法研究[J]. 应用光学,2022,43(1):67-73. WANG Yi, MA Zhengdong, DONG Guanglin. Parts recognition method based on improved Faster RCNN[J]. Journal of Applied Optics,2022,43(1):67-73. doi: 10.5768/JAO202243.0102003
[5]	ZHANG Zhenli, ZHANG Xiangyu, PENG Chao, et al. Exfuse: enhancing feature fusion for semantic segmentation[C]//Proceedings of the European Conference on Computer Vision (ECCV),Munich,Germany:Springer, 2018: 269-284.
[6]	YANG Xue, YAN Junchi, FENG Ziming, et al. R3Det: refined single-stage detector with feature refinement for rotating object[C]//Proceedings of the AAAI Conference on Artificial Intelligence,Palo Alto,California USA:AAAI Press,2021, 35(4): 3163-3171.
[7]	LIN Tsungyi, DOLLAR Piotr, GIRSHICK Ross, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA: IEEE, 2017: 2117-2125.
[8]	LIU Shu, QI Lu, QIN Haifang, et al. Path aggregation network for instance segmentation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),Salt Lake City,UT,USA:IEEE, 2018: 8759-8768.
[9]	柴斌. 基于深度学习的工件检测和定位系统的研究与实现[D]. 沈阳: 中国科学院大学中国科学院沈阳计算技术研究所, 2021. CHAI Bin. Design and research of workpiece defect detection system based on deep learning[D]. Shenyang: Shenyang Insitute of Computing Technology, Chinese Academy of Science. 2021.
[10]	MEHTA Sachin, RASTEGARI Mohammad, CASPI Anat, et al. ESPNet: efficient spatial pyramid of dilated convolutions for semantic segmentation[C]//Proceedings of the European Conference on Computer Vision (ECCV),Munich, Germany:Springer, 2018: 552-568.
[11]	刘宽, 郎磊. 轻量化SSD目标检测方法研究[J]. 湖北民族大学学报(自然科学版), 2021, 39(4): 418-424. LIU Kuan, LANG Lei. Research on lightweight SSD target detection method[J]. Journal Of HuBei Minzu University(Natural Science Edition) , 2021, 39(4): 418-424.
[12]	TAN Mingxing, LE Quoc. Efficientnet: rethinking model scaling for convolutional neural networks[C]//Proceedings of Machine Learning Research(PMLR),Long Beach,California,USA:PMLR, 2019: 6105-6114.
[13]	曹富强, 王明泉, 张俊生, 等. 基于深度学习的铸件X射线图像分割研究[J]. 应用光学,2021,42(6):1025-1033. CAO Fuqiang, WANG Mingquan, ZHANG Junsheng, et al. Casting X-ray image segmentation based on deep learning[J]. Journal of Applied Optics,2021,42(6):1025-1033. doi: 10.5768/JAO202142.0602003
[14]	刘怀广, 丁晚成, 黄千稳. 基于轻量化卷积神经网络的光伏电池片缺陷检测方法研究[J]. 应用光学,2022,43(1):87-94. LIU Huaiguang, DING Wancheng, HUANG Qianwen. Defects detection method of photovoltaic cells based on lightweightconvolutional neural network[J]. Journal of Applied Optics,2022,43(1):87-94. doi: 10.5768/JAO202243.0103003
[15]	卢艳东, 李积英, 王筱婷. 一种基于改进YOLOv3-tiny的轻量级轨道紧固件检测算法[J]. 铁道标准设计, 2021, 14（2）: 1-7. LU Yandong, LI Jiying, WANG Xiaoting. A lightweight track fastener detection algorithm based on improved YOLOv3-tiny[J]. Railway Standard Design, 2021, 14（2）: 1-7.
[16]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, 2016: 770-778.
[17]	HAN Kai, WANG Yunhe, TIAN Qi, et al. Ghostnet: more features from cheap operations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),Seattle,WA,USA:IEEE, 2020: 1580-1589.
[18]	WANG Chienyao, LIAO H M, WU Y H, et al. CSPNet: a new backbone that can enhance learning capability of cnn [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Seattle,WA, USA: IEEE, 2020: 390-391.
[19]	YANG Xue, YAN Junchi. Arbitrary-oriented object detection with circular smooth label[C]//Proceedings of the European Conference on Computer Vision (ECCV),Glasgow,United Kingdom:Springer,2020: 677-694.
[20]	XIA Guisong, BAI Xiang, DING Jian, et al. DOTA: a large-scale dataset for object detection in aerial images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),Salt Lake City,UT,USA:IEEE, 2018: 3974-3983.
[21]	FENG Pengming, LIN Youtian, GUAN Jian, et al. TOSO: student’s-T distribution aided one-stage orientation target detection in remote sensing images[C]// IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),Virtual,Barcelona:IEEE, 2020: 4057-4061.
[22]	CHEN Zhiming, CHEN Kean, LIN Weiyao, et al. PIoU loss: towards accurate oriented object detection in complex environments[C]//Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, United Kingdom: Springer, 2020: 195-211.
[23]	XIAO Zhifeng, QIAN Linjun, SHAO Weiping, et al. Axis learning for orientated objects detection in aerial images[J]. Remote Sensing, 2020, 12(6): 908-928.
[24]	CAO Lianyu, ZHANG Xiaolu, WANG Zhaoshun, et al. Multi angle rotation object detection for remote sensing image based on modified feature pyramid networks [J]. International Journal of Remote Sensing,2021, 42(14): 5253-5276.
[25]	LI Wei, WEI Wei, ZHANG Lei. GSDet: object detection in aerial images based on scale reasoning[J]. IEEE Transactions on Image Processing, 2021, 30: 4599-4609.
[26]	LI Yangyang, HUANG Qin, PEI Xuan, et al. RADet: refine feature pyramid network and multi-layer attention network for arbitrary-oriented object detection of remote sensing images [J]. International Journal of Remote Sensing,2020, 12(3): 389-408.
[27]	DING Jian, XUE Nan, LONG Yang, et al. Learning roI transformer for oriented object detection in aerial images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),Long Beach,CA,USA:IEEE, 2019: 2844-2853.
[28]	YI Jingru, WU Pengxiang, LIU Bo, et al. Oriented object detection in aerial images with box boundary-aware vectors[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA,: IEEE, 2021: 2149-2158.
[29]	YANG Xue, YANG Jirui, YAN Junchi, et al. SCRDet: towards more robust detection for small, cluttered and rotated objects[C]//Proceedings of the IEEE International Conference on Computer Vision(ICCV), Seoul, Korea(South): IEEE, 2019: 8231-8240.
[30]	LI Chengyuan, LUO Bin, HONG Hailong, et al. Object detection based on global-local saliency constraint in aerial images[J]. International Journal of Remote Sensing, 2020, 12(9): 1435-1456.
[31]	WANG Mengyuan, ZHANG Xuanyu, YU Chuanbo, et al. Oriented object detection with fine-grained enhancement and angle constraint[C]//Proceedings of the International Conference on Computer Science & Education (ICCSE),Lancaster University,UK:IEEE,2021: 752-757.
[32]	KIM Beomyoung, LEE Janghyeon, LEE Sihaeng, et al. TricubeNet: 2D kernel-based object representation for weakly-occluded oriented object detection[C]//Proceedings of the IEEE Winter Conference onApplications of Computer Vision(WACV), Waikoloa, HI, USA: IEEE, 2022: 167-176.
[33]	GUO Zonghao, LIU Chang, ZHANG Xiaosong, et al. Beyond bounding-box: convex-hull feature adaptation for oriented and densely packed object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Nashville, TN, USA: IEEE, 2021: 8792-8801.