连续时间马氏决策过程均值-方差优化问题的研究

In this project, we consider a series of mean-variance optimality problems for continuous-time Markov decision processes (MDPs). The main object is to obtain some policies that minimize the variance over a set of all policies with a given expected reward, which satisfies suitable conditions. We are planning to solve the three following questions: .1) For Markowitz mean-variance models, we aim to find a policy that minimizes the variance over a set of all policies with a optimal/given expected reward. Using the conditional expectation and Markov property we can prove that the mean-variance optimality problem can be transformed to an equivalent discounted optimality problem, and establish the existence of mean-variance optimal policies. Furthermore, we will analysis their computational methods..2) We establish the constrained continuous-time MDPs models with mean-variance optimality criterion. Using the theory of constrained continuous-time MDPs, we will find the condition of existence of constrained mean-variance optimal policies and their computational methods..3) We first introduce the concept of risk-averse dynamic programming, and employ the Markov risk measures. Using these new tools，we will establish the existence of discounted/average/mean-variance optimal policies. Moreover, we will analysis the difference and relationship between this new method and the classical methods used in optimality problems for MDPs, such that MDPs can be applied for more real situations..The studies of the above questions are original and the first time appeared in the relative research areas.

本项目主要研究连续时间马氏决策过程的均值-方差优化问题。拟解决以下三个问题：1）针对Markowitz均值-方差模型，在期望折扣收益最大化或等于某个给定常数的前提下，寻找相应方差最小的策略。通过分析其与折扣准则的理论关系，得到均值-方差最优策略存在的条件，进而得到其计算方法；2）通过建立受约束连续时间MDP均值-方差模型，在期望收益不小于给定常数的条件下，寻找使方差达到最小的策略。运用受约束模型的已有理论结果，分析当前准则下最优策略的存在性以及它的计算方法；3）利用风险中立动态规划新方法，处理连续时间MDP中的折扣、平均和均值-方差最优化问题，建立最优策略存在性，进一步分析相应的计算方法。另外，将分析该方法与现有理论方法的区别和联系，从而扩大MDP的应用范围。以上三个问题的研究均是首次的。

本项目致力于连续时间马氏决策过程(MDP)均值方差优化准则和受约束MDP以及Markov对策（博奕）的研究。主要成果如下：.1．首次建立Polishh空间下连续时间MDP均值-方差准则的最优方程，给出了最优性条件和应用例子。该成果巧妙地建立了均值-方差准则与期望折扣准则的“等价”关系，从而得到最优策略的存在性，完善了连续时间MDP在折扣模型下的均值-方差准则相关理论结果。.2. 对受约束的连续时间MDP，首次讨论了模型收敛性问题。该成果首次证明了连续时间MDP的有限状态模型序列中的（约束）最优策略与最优值函数收敛到无限状态模型中对应的（约束）最优策略与最优值函数，并构造出有限状态模型来逼近可数状态模型。.3. 对状态空间可数、费用率函数无界的非零和受约束离散时间Markov对策，提出了“双逼近方法”，在较弱的条件下证明了约束随机对策的Nash平衡点的存在性。其中第二个方法是新方法：构造一系列辅助对策模型，证明该辅助对策模型的Nash平衡点序列收敛到原约束对策模型的Nash平衡点。.4. 对于受约束的MDP，首次给出了首达目标准则下最优策略存在性。该成果填补了受约束MDP中首达目标准则理论的空白，不仅提出了若干新的最优性条件，且当中的折扣因子具有一般性（同时依赖于状态和行动）。