大数据环境下的文本挖掘并行处理技术研究

In recent years, text data in a big data environment has the characteristics of large amount of data, wide distribution, dynamic growth, high real-time requirements, and high noise, and has become a hot research object of recent text mining parallel processing techniques. However, text mining parallel processing techniques in a big data environment has the problems of low performance in parallel performance, lack of real-time performance, and low accuracy. Therefore, how to design an efficient parallel processing scheme in a big data environment to improve text mining efficiency and accuracy is a very important issue. This proposal focuses on the key problem of "improving efficiency and accuracy", and analyzes the existing problems of parallel processing techniques of text mining in existing big data environments. In this proposal, firstly, we construct a big data processing platform based on a heterogeneous CPU-GPU cluster and use its CPU and GPU cooperative computing capability, consider load balancing and data communication issues, establish a more general strategy of algorithmic parallelism and hardware collaboration. Secondly, according to requirements such as real-time applications, the research on parallel generation algorithm based on feature engineering synchronization is performed to further achieve the purpose of improving text mining performance. Finally, on the premise of ensuring the accuracy of mining, a parallel algorithm based on deep learning noise filtering is designed. The research of the subject will provide an important reference for the future text mining parallel processing technology.

近年来，大数据环境下的文本数据具有数据量大、分布广泛、动态增长、实时性要求高、噪声多等特点，成为近期文本挖掘并行处理技术的热点研究对象。然而，大数据环境下的文本挖掘并行处理技术具有性能不高、实时性得不到满足、挖掘准确性低等问题。因此，如何在大数据环境下设计高效的并行处理方案来提高文本挖掘效率与准确性是一个非常重要的问题。本课题着眼于“提高效率与准确性”这一关键问题，分析现有大数据环境下文本挖掘并行处理技术存在的问题，首先构建异构CPU-GPU集群大数据处理平台，利用其CPU-GPU协作计算能力，结合考虑负载均衡和数据通信问题，确立更具一般性的算法并行与硬件协同的策略；其次根据应用实时性等需求，展开基于特征工程同步并行生成的算法研究，以达到进一步提高文本挖掘性能的目的。最后，在保证准确性的前提下，设计基于深度学习噪声过滤的并行算法。课题的研究为未来文本挖掘并行处理技术提供重要的借鉴。

“大数据环境下的文本挖掘并行处理技术研究 ”项目组针对大数据环境下的文本 挖掘并行处理技术具有性能不高、实时性得不到满足、挖掘准确性低等问题。以如何在大数据环境下设计高效的并行处理方案来提高文本挖掘效率与准确性为研究目标，分析现有大数据环境下文本挖掘中情感识别、实体命名识别和文本去重技术等关键处理技术中存在的问题，着眼于“提高效率与准确性”，首先为情感识别任务设计出一种多元消息传递模型，并基于不同属性的异质性来构建图神经网络来模拟说话者级别之间的交互关系，来降低模型的复杂度，使模型具有更好地鲁棒性；其次提出了采用多模态数据集提取对话文本的不同特征，并引入自注意力机制的多模态情感识别，从而提高了识别准确率和性能。同时，提出了一种基于图节点选择和节点优化策略的两阶段主体事件去重方法，实现更有效和高效去重。最后基于企业成分优化选择和单字符优化选择的双通道企业简称自动生成，提高了企业实体识别的准确率和性能。通过三年的研究工作，项目组完成了各项研究内容，形成了多样的高效的并行处理方案来提高文本挖掘效率与准确性，为未来文本挖掘中如何“提高效率与准确性”提供了重要的借鉴。