基于博弈强化学习的交通态势协同演化机制研究

In transportation systems, there exist complicatedly-coupled relationships between the dynamic individual routing decisions of vehicles and the changing of traffic conditions, which results in inaccurate expectation for the performance of traffic control and path planning. Although conventional multi-agent based self-organizing evolutionary game can achieve a better solution while considering individual differences and scheduling goal, the system stability is uncontrollable because of the large quantity of vehicles in the actual transportation system. Therefore, the multi-agent based method can be hardly applied to the analysis of global traffic situation evolution. Existing solutions to traffic situation evolution control cannot deal with both efficiency and accuracy, and can hardly be put into practice. To this end, this project takes advantage of the spatiotemporal correlations among vehicles to partition road networks, and then transforms the analysis of traffic situation evolution into the multi-agent evolutionary game and the analysis of traffic situation equilibrium within the regional road networks, which reduces the state space of the original problem and improves the performance of the traffic situation evolution control. Then, as for the complicatedly-coupled relationship between the regional and global traffic situation, this project leverages deep reinforcement learning, which performs multiple iterations and selections to the evolutionary game results in regional road network, to learn the policy for evaluating whether the traffic situation evolves in equilibrium. To achieve collaborative evolution of regional and global traffic situation, game-theoretic reinforcement learning is used to accurately prune the results of regional evolutionary gaming candidates. In this way, this mechanism can make individual preference involved evolution analysis of global traffic situation more efficient and practical.

道路交通环境中，车辆个体的动态选路决策与交通状态变化存在复杂耦合，导致交通调控和驾驶规划效果难以准确预期。传统一般采用多智能代理自组织演化博弈方法，考虑个体差异和调度目标，以获得较优解。但由于实际交通环境中车辆规模大，系统稳定时间不可控，难以适应全局交通态势的演化分析。本项目针对当前交通态势演化方案难以兼顾时效性和准确性，实施困难的问题，拟利用车辆的时空相关性划分交通路网，将交通态势的演化转换为局部路网内多智能代理的演化博弈过程和局部路网间交通流态势均衡过程，降低问题的状态空间。同时，针对由此带来的局部交通态势与全局交通态势复杂耦合问题，拟提出使用深度强化学习，通过对局部路网演化博弈结果的多次迭代筛选，学习交通态势是否均衡发展的评价策略。进而在服务过程中，利用博弈强化学习对局部博弈候选结果准确剪枝，完成局部与全局交通态势的协同演化，实现考虑个体偏好的全局交通态势高效演化分析。

道路交通环境中的车辆个体动态选路决策与交通状态存在复杂耦合，传统方法难以结合车辆个体差异和交通调度目标，导致系统稳定时间不可控，难以保障全局交通态势演化分析的时效性和准确性。本项目主要针对传统方法存在的选路决策用户偏好认知问题、局部交通态势演化预测问题、全局交通态势推演高效性问题展开研究，引入价值迭代网络学习用户个体驾驶行为偏好，利用演化博弈将用户群体选路的集中规划转化为分布式协同规划，通过分层框架将大规模用户选路分解为局部协同规划和基于全局态势演化分析的局部引导策略评价，从而在考虑用户行为偏好条件下实现了大规模交通态势的高效演化。考虑到未来交通环境中，车与路控等交通终端与基础设施的人工智能化发展趋势，本项目进一步提出将多车路群智协同过程建模为多智能体系统，通过博弈强化学习方法训练多智能体协同决策。首先通过注意力机制和时空图卷积网络进行多智能体协同认知，通过价值迭代网络进行态势认知，解决多车路群体智能协同条件下的选路规划问题。之后，利用注意力邻接采样机制提高多智能体系统的观测能力并发现协同意图，通过多智能体深度强化学习训练车路群体智能的演化博弈决策过程，解决了局部交通演化面临的群体自主选路规划与交通态势耦合问题。最后，针对局部交通规划与全局交通态势耦合问题，引入蒙特卡洛树搜索方法对不同局部规划策略可能引发的交通状态进行动态访问和演化评估，通过快速剪枝缓解维数灾难，在实现更高全局交通态势推演搜索效率同时，筛选出区域最优策略并实现大规模交通路径规划控制。本项目提出的博弈强化学习体系符合未来智能化交通场景下的车路群智协同选路规划特征，能够为未来智能交通系统构建出一种既满足协同需求和用户选路偏好，又满足道路交通调控需求，具有准确性和时效性保障的高效交通态势演化分析与选路控制机制，为未来智能交通系统的建设提供指导和保障。