基于深度学习的智能草图三维角色建模

With the rapid development of animation, games and virtual reality industries, the demand for 3D content has increased dramatically. However, it is very challenging for modeling using existing tools, especially for amateur users. For example, creating a simple 3D animal model, such as a horse, often requires a professional artist who has been trained for two or three years to spend 3-4 hours. .This proposal aims to introduce advanced deep learning techniques to update traditional interactive modeling design. And we plan to develop a real-time sketching system for 3D character modeling, which allows an amateur user evenly without drawing skills to create vivid models in few minutes. Considering the differences between the head and body part of the character, we will study how to model them separately. For the head part, we intend to design a mesh deformation based deep neural network. It first extracts geometric-aware features from the input sketch and then maps them onto a template mesh. After that, we learn a deformation mechanism to make the deformed mesh to match the shape of the input sketch. For the body part, we plan to firstly design a novel interactive method based on drawing arcs along the bones, to determine the three-dimensional skeleton. Then a rough multi-limb mesh can be directly generated. Later, a self-supervised learning approach will be applied to transform the rough model to fit the user's 2D contour. In the end, we combine these two parts and complete the final interactive interface design.

随着动画、游戏以及虚拟现实产业的快速发展，业界对于三维内容的需求也日益剧增。然而基于现有的工具，要想得到一个相对精细的三维模型往往需要一个培养了两三年时间的专业建模师，且需要大量的时间成本。本课题旨在将先进的深度学习技术引入到交互建模设计中，并开发一套可以让普通用户也能够使用的基于草图的实时三维角色建模系统。我们针对不同部位的特点，将针对角色的头部与身体分别进行研究。对于头部部位，我们拟设计一套基于模板网格变形的深度学习框架，使其首先从草图中提取重要的几何特征并映射到模板网格上以学习出一个变形的机制以匹配草图的轮廓。而对于身体部位，我们拟首先设计一个基于画弧线簇的交互方式进行三维骨架的确定，基于此一个粗糙的多肢体网格将被生成出来。之后，一个自监督的学习方式将被应用于如何将粗糙的模型变形以拟合用户画的二维轮廓。最终，我们将这两部分结合并完成完整的交互界面设计。

三维角色建模在动画、游戏以及元宇宙等行业有着非常重要的作用，然而传统建模师往往需要花费昂贵的时间进行精细的建模。本项目主要围绕基于深度学习技术的智能交互式三维头部角色建模展开，主要目的为构建高效易用的基于草图的交互式建模系统。从系统方面，我们构建了世界上首个基于草图的三维动物头角色建模系统。该系统可以支持业余用户进行快速的三维角色造型，其在动画行业以及教育行业有很大的应用前景；从技术方面，我们基于该具体应用，支持了前沿的三维深度学习基础研究。具体来说，针对如何利用深度学习技术进行三维网格的生成，我们三年内进行了多种技术的探索。包括基于图卷积神经网络的网格变形技术和基于隐式表达引导的显式网格造型技术；从数据方面，我们构建了世界上首个三维动物头数据集以及三维夸张人脸数据集，并被多家研究机构关注与使用。同时，基于该项目中研究的基于深度学习的三维网格生成技术，我们成功的应用到了多种场景的三维生成任务中，如三维人脸、三维物体与场景以及三维衣物等。