基于迁移学习的多模态信息协同目标感知

Cooperative objects perception based on multi-modal information is an important part in the field of multi-media sensor network, and is also one of the difficult issues arising from the integration of multi-media information fusion and objects tracking theory. Different sensor properties and measurement relationship of acoustic and video sensor in multi-media sensor network are analyzed. By digging through explicit and non-explicit relationship of homogeneous and heterogeneous sensing data, transfer learning theory is studied to complete data transformation in both temporal and spatial space, which will improve the environmental perception and self-healing performance of the network. Temporal and spatial distribution characteristics of audio and video sensors are studied to build the intra-layer cross information fusion and inter-layer mutual propagation fusion, which helps to realize the global optimization of energy consumption and perception performance. Multi-view targets feature representation of the multi-modal information is studied to establish the multi-modal data fusion strategies in associated layer to improve the accuracy of the data association. Relationship and transformation of heterogeneous sensor data is studied to establish features transform and sharing mechanism, which will improve objects matching performance of the collaborative sensing system. Potentially temporal and spatial relevant characteristics are studied to establish tracking data transform between non-neighboring clusters, which helps to fill network sensing holes. Experimental platform will be established, in which verification and evaluation of the proposed algorithms are completed from multiple perspectives. The project belongs to basic research for application, and has broad application prospects in industrial, civilian, military and other fields.

多模态信息协同目标感知是多媒体传感器网络研究的重要部分，也是随着多媒体信息融合理论在目标跟踪领域不断深入而产生的难点问题之一。课题分析多媒体传感器网络中音视频的传感特性及量测相关，研究迁移学习理论，挖掘同质与异质感知信息的显式与非显式相关，实现多模态数据的时空迁移，提高网络环境感知性能。研究音视频传感数据的时空分布特性，构建层内多传感信息交叉、层间融合信息双向传播的融合体系，实现网络能耗与性能的整体优化；研究多模态特征信息的多视图表示，建立关联层多模态数据融合策略，提高数据关联的准确性；研究异类传感数据的关联与转换，建立异类特征的迁移与共享机制，提高协同感知系统的目标匹配性能；研究传感信息的潜在时空关联特性，建立非邻簇点跟踪数据迁移机制，填补网络空洞；以算法验证为目标构建实验平台，从多角度完成所提方案的验证和评价。本项目属于应用基础研究，研究成果在工业、民生、军事等领域有广泛的应用前景。

课题针对多媒体传感器网络中的多模态信息目标协同感知问题，从构建层内多传感信息交叉、层间融合信息双向传播的融合体系出发，分析同质信息和异质信息的量测特点，以迁移学习和联合学习的形式，实现多模态数据的时空融合，提高网络环境感知性能。在数据关联层，课题研究多模态数据关联策略，分别以生成对抗学习和线性判别分析为基础，探索多模态数据关联的多视图表示，通过将目标的图像特征和音频或文本特征映射到联系更加紧密的公共空间，实现多模态数据之间的关联；课题将数据关联方法应用于说话人识别和多模态检索任务中，实验表明，课题所建立的多模态数据关联特征空间可有效提高目标类型判别精度，相较于近年方法提高近3.0%。在目标识别层，课题研究同质异类数据以及异质数据的共通性和互补性，通过建立异类信息的迁移和共享机制，提高协同感知系统的目标匹配及识别能力；课题将异质融合方法应用于说话人识别，将同质异类融合方法应用于行人再识别、人脸重建及姿态估计，实验表明，相比于近年方法，课题方法在说话人识别上精度提高约2.5%，在行人再识别上精度提高约3%，在人脸重建上精度提高约2.5%，在姿态估计上精度提高约3.5%。在目标跟踪层，首先设计单传感下面对复杂环境的目标跟踪算法，并通过研究传感信息的时空关联特性，学习多摄像头之间的相似性度量，建立跟踪目标的风格迁移机制，实现稀疏节点环境下的多摄像头目标跟踪；实验表明，相比于近年方法，课题所研究的单传感目标跟踪方法精度提高约5.0%，多摄像头目标跟踪精度提高约6.0%。课题共发表学术论文49篇，其中SCI论文6篇，EI论文21篇；项目研究成果服务本地物联网产业，部分实现项目的成果转化工作，申获发明专利3项，申请发明专利8项。课题研究既在理论层面上建立了迁移学习机制下多媒体传感器网络多传感信息共享的系统性研究体系，又为基于迁移学习的目标感知算法向实用性技术的转化提供有效方案。