基于预测信息的智能体决策模型学习及应用研究

For agents operate in an environment, how to act according to an optimal policy is a central problem in artificial intelligence. To solve the decision making problem, one commonly used technique is by modelling dynamical systems firstly and then solving the problem using the learned model. However, current research on this topic usually assumes the learned model is completely correct or by making some strong assumptions on the parameters of the model, which is contrary to the fact that in practice the learned model can be hardly exact. Predictive state representations (PSRs) offer an expressive framework for modelling dynamical systems. By compactly representing the state as functions of observable predictions, compared to the latent-state based approaches, e.g. partially observable Markov decision processes, the PSR model is easier to learn and less prone to local minima problems. In this project, how to make decisions with the learned model in large systems by utilizing the advantages and distinguishing features of PSRs is studied. Firstly, by studying the mechanism of the state representations and combing the included prediction information in the representation, a task-based state representation with the purposed of improving the quality of decision making is proposed. Accordingly, the techniques for learning the PSR model with the corresponding state representation will be studied. Then, with the advantage of Monte-Carlo tree search for searching solutions in large scale problems, by using the prediction information provided by the learned model as a guidance, a decision making approach based on the learned model will be proposed. Simultaneously, the learned model can be online updated in the process of searching the decision solutions. With the success of the proposed project, decision making based on the learned model will be applied into the large scale systems.

对动态系统建模，进而根据模型求解智能体决策策略是人工智能研究的基本问题。现有基于模型的决策算法，通常假定模型完全准确或对模型参数设定较强假设条件。预测状态表示（PSRs）是对动态系统建模的有效方法，该方法用完全可观测的预测信息表示系统状态，相比基于隐状态的建模方法，例如POMDP等，模型较易学习、不易陷入局部极小点。本项目拟以大规模系统为应用对象，利用PSRs的优势和特点，为基于获取模型的决策研究提供解决方案。首先研究PSRs状态表示机制，结合状态表示中的预测信息，提出“任务”驱动的同决策结合的状态表示方式；然后根据所提状态表示特点，研究相应模型获取方案；进而，根据模型提供的预测信息，利用Monte-Carlo 树搜索(MCTS)的优势，结合MCTS，实现基于获取模型的智能体决策算法，并同时实现模型的在线更新。通过项目的开展，可望实现基于获取模型的决策算法在大规模系统中的应用。

项目主要开展动态环境下智能体的决策研究，重点聚焦于动态系统模型的建立，以及基于获取模型的规划和智能体的决策性能、效率的提升。.在获批项目资助下，主要针对以下内容进行了研究：1）动态系统的模型建立。研究了模型熵同模型获取精度之间的关系，进而提出了基于模型熵的动态系统模型建立策略，同时针对连续状态输入环境，实现了动态系统模型的建立及同深度强化学习有关算法的结合；2）基于获取模型的规划。仅依据训练数据，无需先验知识，通过利用模型预测信息，结合Monte-Carlo 树搜索，实现了从“零”开始的模型的获取和基于模型的离线、在线规划，并理论上证明了有关方案的优势和收敛性； 3）模型决策性能、效率的提升。研究了高维输入状态下，通过利用、改进Attention机制、对比学习、输入帧序列间的时序信息，提升智能体的决策性能, 并进一步探讨了局部可观测、奖励延迟、稀疏等实际场景下智能体的决策、探索策略；4）经验数据的有效再利用。如何有效再利用已有经验数据是当前研究的重要内容，项目开展过程中，通过提取的基于真实数据的未来信息用于智能体的决策过程及利用主动优化方式，主动从情景记忆中快速锁定与当前类似的历史良好情景，并使之再次用于监督当前策略的优化，进一步提高了智能体的最终执行性能；5）其他。基于智能体决策的研究成果，提出了基于蒙特卡洛树搜索的特征选择算法等其它算法。.通过项目的开展，拓展了有关方法的应用规模，对于离散输入状态环境，实现了较大规模系统模型的获取和基于模型的规划；针对高维输入环境，研究方案在典型基准问题上，例如Atari 2600 games，DeepMind Control Suite，OpenAI Gym等，做了验证；同时，研究方案在脓毒症患者的治疗上，在包含17,898位脓毒症患者信息的MIMIC-III数据集上验证了所提方案的有效性，性能相比基准实现了显著提升。.智能体的决策规划是人工智能研究的基本问题，有关研究具有重要的理论意义和应用前景。项目的开展推动了模型学习、基于获取模型的决策的研究，为有关决策问题提供了有效解决方案。