Item2vec与改进DDPG相融合的推荐算法

Item2vec与改进DDPG相融合的推荐算法 Recommendation Algorithm Combining Item2vec with Improved DDPG 1 first-author 刘文竹，女，硕士生，主要研究方向为人工智能、大数据分析。E-mail：wenzhul@163.com 刘文竹，女，硕士生，主要研究方向为人工智能、大数据分析。E-mail：wenzhul@163.com wenzhul@163.com 1 2 corresp E-mail：huangbosues@sues.edu.cn E-mail：huangbosues@sues.edu.cn huangbosues@sues.edu.cn 1 1 1 3 上海工程技术大学 电子电气工程学院，上海 201620 上海工程技术大学 电子电气工程学院，上海 201620 School of Electronic and Electrical Engineering，Shanghai University of Engineering Science，Shanghai 201620，China School of Electronic and Electrical Engineering，Shanghai University of Engineering Science，Shanghai 201620，China 江西省经济犯罪侦查与防控技术协同创新中心，江西 南昌 330103 江西省经济犯罪侦查与防控技术协同创新中心，江西 南昌 330103 Center of Economic Crime Detection and Prevention and Control Technology Collaborative Innovation，Nanchang 330103，Jiangxi，China Center of Economic Crime Detection and Prevention and Control Technology Collaborative Innovation，Nanchang 330103，Jiangxi，China 上海外国语大学 国际金融贸易学院，上海 201620 上海外国语大学 国际金融贸易学院，上海 201620 School of Economics and Finance，Shanghai International Studies University，Shanghai 201620，China School of Economics and Finance，Shanghai International Studies University，Shanghai 201620，China 传统的推荐算法能够有效解决信息过载问题，但在冷启动和数据稀疏的情况下，传统方法仍有其局限性。针对以上问题本文提出一种基于深度强化学习理论的推荐算法，该算法使用深度确定性策略梯度（DDPG，deep deterministic policy gradient，DDPG）算法来解决推荐问题，使用Item2vec将离散的动作空间转换为连续的表示，同时提出了一种余弦距离和欧氏距离相结合的奖励函数，能够保障神经网络不会过早的收敛于局部最优。应用该算法进行电影的推荐，实验结果表明本文提出的算法能够产生较好的推荐并能缓解冷启动所带来的影响。 The traditional recommendation algorithm can effectively solve the problem of information overload, but in the case of cold start and data sparse, the traditional method still has its limitations. For these problems, this paper proposed a recommendation algorithm based on deep reinforcement learning theory, which using the deep deterministic policy gradient (DDPG) algorithm to solve the recommendation problem and using Item2vec to transform the discrete action space into a continuous representation. A reward function combining cosine distance and Euclidean distance is proposed to ensure that the neural network does not converge to local optimum prematurely. This paper applied the algorithm to movie recommender system. The final experiment proved that the proposed algorithm can generate better recommendation results and alleviate the impact of cold start. 推荐系统 深度强化学习 冷启动 recommender system deep reinforcement learning cold start 刘文竹,黄勃,高永彬, 等.Item2vec与改进DDPG相融合的推荐算法[J].武汉大学学报（理学版）,2019,65(3):297-302. DOI:10.14188/j.1671-8836.2019.03.010. 0　 引 言 1 推荐系统（recommender system）作为一种高效的信息过滤技术，受到了广泛的关注。推荐系统挖掘用户偏好信息，对用户兴趣进行建模，从而为用户推荐可能满足其需求的物品。传统的推荐算法可以分为4类：协同过滤推荐算法 ［ 1 1 ］ 、基于内容的推荐算法、基于知识的推荐算法和混合推荐算法 ［ 2 2 ］ 。 近年来，推荐系统领域也产生了许多非传统的推荐算法。许多公司使用基于深度神经网络、多臂赌博机（multi-armed bandit）等推荐算法，希望能够提高推荐质量。Okura等 ［ 3 3 ］ 为雅虎新闻提出了基于RNN（recurrent neural network）的新闻推荐系统；Covington等 ［ 4 4 ］ 提出基于深度神经网络的推荐算法，为YouTube解决视频推荐问题。 现有的推荐算法多将推荐看作一个静态过程，由此产生固定的推荐策略，这样的推荐策略无法实时适应用户兴趣的改变，对兴趣变化较快的用户很难产生实时性的推荐 ［ 5 5 ］ ，同时对于推荐系统中始终存在的数据稀疏性问题也没有很好地解决。因此，有些学者尝试采用强化学习 ［ 6 6 ］ （reinforcement learning，RL）技术来解决现实场景中的推荐问题。 强化学习是机器学习领域一个重要的研究热点。Shani等 ［ 7 7 ］ 认为产生推荐的过程是一个连续决策的问题，于2005年提出可以将推荐问题看作一个马尔可夫决策过程（Markov decision process，MDP）。Taghipour等 ［ 8 8 ］ 在2008年提出将网页信息与Q学习算法结合解决网页推荐问题，但这种方法未能解决状态空间和动作空间过大的推荐任务。为了控制状态空间和动作空间的大小，Choi等 ［ 9 9 ］ 提出一种基于双聚类的Q学习算法，从用户和项目两个方面对用户-项目矩阵（user-item matrix）进行双聚类，然后再使用Q学习算法产生推荐。但是这种方法仅仅在一定程度上缓解了状态空间、动作空间较大的MDP问题。有研究者在Q学习中引入值函数估计（value function approximator），提出Deep Q-Net（DQN）方法 ［ 10 10 ， 11 11 ］ ，但DQN 是一个面向离散动作空间的算法，而现实中许多问题是连续的，比如控制问题等，此外Q学习算法和DQN算法都是Value-based的学习方法，通过对Bellman方程进行迭代最终收敛到最优价值函数，这种方法计算量大，而且在一些特殊的场景下Q值难以计算。Policy-based ［ 6 6 ］ 的学习方法——策略梯度（policy gradient）则不存在这样的问题，这种方法可以直接对策略进行学习，但基于策略的方法只能进行回合更新，收敛速度较慢。Sliver等 ［ 12 12 ］ 结合Value-based和Policy-based的方法提出了Actor-Critic框架，但Actor的更新取决于Critic的价值判断，而Critic难收敛， 再加上Actor的更新，导致Actor-Critic可能无法收敛。Lillicrap等 ［ 13 13 ］ 为了解决Actor-Critic存在的收敛问题，在此基础上融入DQN的优势，提出了深度确定策略梯度（deep deterministic policy gradient，DDPG）算法。Zhao等 ［ 14 14 ， 15 15 ］ 使用Actor-Critic框架，采用DDPG算法来解决电商中的推荐问题。Munemasa等 ［ 16 16 ］ 使用隐狄利克雷分布（latent Dirichlet allocation，LDA）将商铺信息转化为分布式表示（distribute representation），然后再采用DDPG算法解决商铺推荐问题。本文提出一种基于DDPG强化学习算法来解决现实场景中的推荐任务，使用Item2vec ［ 17 17 ］ 将离散的电影项目转化为连续的词向量表示，通过DDPG算法学习最优推荐策略，解决电影推荐问题。 1　 问题描述和定义 1 本文融合Item2vec和改进的DDPG算法来解决推荐问题，首先将推荐问题归纳为一个马尔科夫决策过程，如 图1 图1 所示，在 t 时刻，Recommender Agent 接收用户此刻的状态 <math id="M1"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206774&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206769&type= 2.03200006 3.72533321 并根据该状态产生推荐 <math id="M2"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206783&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206779&type= 2.62466669 3.72533321 ，user接收推荐 <math id="M3"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206792&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206788&type= 2.62466669 3.72533321 ，推荐系统进入 t+ 1时刻，user产生推荐对应的奖励同时状态更新再次传送给Recommender Agent。一般地，每一个马尔科夫决策过程由一个五元组组成（ S ， A ， P ， R ， γ ），以下为五元组各个元素在推荐系统中的定义 ［ 14 14 ］ ： <math id="M4"><mi>S</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206811&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206807&type= 2.20133328 2.79399991 是所有环境状态的集合， <math id="M5"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>∈</mo><mi>S</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206821&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206815&type= 8.21266651 4.06400013 表示Agent在 t 时刻所处的状态。本文使用电影网站movielens数据集进行测试，定义 <math id="M6"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>=</mo><mfenced open="{" close="" separators="|"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>t</mi></mrow><mrow><mn mathvariant="normal">1</mn></mrow></msubsup></mrow></mfenced><mo>,</mo><mo>…</mo><mfenced open="" close="}" separators="|"><mrow><mo>,</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>t</mi></mrow><mrow><mi>K</mi></mrow></msubsup></mrow></mfenced></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206828&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206826&type= 39.87800217 4.48733330 ，表示用户在 t 时刻的前 <math id="M7"><mi>K</mi><mi mathvariant="normal">个</mi><mo stretchy="false">(</mo><mn mathvariant="normal">1</mn><mo>≤</mo><mi>K</mi><mo>≤</mo><mn mathvariant="normal">5</mn><mo>,</mo><mi>K</mi><mo>∈</mo><msup><mrow><mi mathvariant="normal">ℕ</mi></mrow><mrow><mi mathvariant="normal">*</mi></mrow></msup><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206838&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206833&type= 37.67666626 3.97933340 具有历史评分的电影。 <math id="M8"><mi>A</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206849&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206843&type= 2.87866688 2.79399991 是智能体（Agent）可执行动作的集合，在本文中 A 为所有可以推荐给用户的电影集合。 <math id="M9"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>∈</mo><mi>A</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206859&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206854&type= 9.56733322 4.06400013 表示Agent在 t 时刻所采取的动作， <math id="M10"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206871&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206866&type= 2.62466669 3.72533321 在本文中表示为Agent根据当前的 <math id="M11"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206878&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206873&type= 2.03200006 3.72533321 为用户推荐的电影。 R ： <math id="M12"><mi>S</mi><mo>×</mo><mi>A</mi><mo>→</mo><mi>R</mi><mi mathvariant="normal">是</mi><mi mathvariant="normal">奖</mi><mi mathvariant="normal">励</mi><mi mathvariant="normal">函</mi><mi mathvariant="normal">数</mi><mo>,</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206889&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206884&type= 37.42266846 3.55599999 是用户针对Agent在 t 时刻推荐的电影 <math id="M13"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206899&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206895&type= 2.62466669 3.72533321 所给予的反馈。例如，在 t 时刻，对于Agent推荐的电影，用户会产生不同的反馈，此时Agent就根据用户的反馈获得立即奖励 <math id="M14"><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206909&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206904&type= 12.44600010 4.48733330 。 P 是状态转移概率， <math id="M15"><msubsup><mrow><mi>P</mi></mrow><mrow><mi>s</mi><mi>s</mi><mi>'</mi></mrow><mrow><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></mrow></msubsup><mo>=</mo><mi>P</mi><mo stretchy="false">[</mo><msub><mrow><mi mathvariant="normal">S</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn mathvariant="normal">1</mn></mrow></msub><mo>=</mo><mi>s</mi><mi mathvariant="normal">'</mi><mo stretchy="false">|</mo><msub><mrow><mi>S</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>=</mo><mi>s</mi><mo>,</mo><msub><mrow><mi>A</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">]</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206919&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206914&type= 52.49333191 4.99533367 ，其中 <math id="M16"><msub><mrow><mi mathvariant="normal">S</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206927&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206924&type= 3.04799986 3.72533321 表示 t 时刻的状态， <math id="M17"><msub><mrow><mi mathvariant="normal">S</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn mathvariant="normal">1</mn></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206938&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206932&type= 6.68866634 3.72533321 表示 t +1时刻的状态， <math id="M18"><msub><mrow><mi>A</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206949&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206944&type= 3.64066648 3.72533321 为 <math id="M19"><mi>t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206961&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206954&type= 1.10066664 2.79399991 时刻用户执行操作的项目。如果用户进行了相关操作，环境就对状态进行更新，获得新状态 <math id="M20"><msup><mrow><mi>s</mi></mrow><mrow><mi mathvariant="normal">'</mi></mrow></msup><mo>=</mo><mfenced open="{" close="" separators="|"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>t</mi></mrow><mrow><mn mathvariant="normal">2</mn></mrow></msubsup></mrow></mfenced><mo>,</mo><mo>…</mo><mfenced open="" close="}" separators="|"><mrow><mo>,</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>t</mi></mrow><mrow><mi>K</mi></mrow></msubsup><mo>,</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></mrow></mfenced></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206969&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206966&type= 44.36533356 5.24933338 。 <math id="M21"><mi>γ</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206979&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206975&type= 1.94733334 3.64066648 是折扣因子， <math id="M22"><mi>γ</mi><mo>∈</mo><mo stretchy="false">[</mo><mn mathvariant="normal">0,1</mn><mo stretchy="false">]</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206990&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206985&type= 14.64733315 3.72533321 ， 用来权衡未来奖励对累积奖赏的影响。当 <math id="M23"><mi>γ</mi><mo>=</mo><mn mathvariant="normal">0</mn></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206999&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4206994&type= 8.63599968 3.64066648 时，推荐系统只关注立即奖励；当 <math id="M24"><mi>γ</mi><mo>=</mo><mn mathvariant="normal">1</mn></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207013&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207003&type= 8.63599968 3.64066648 时，推荐系统关注长期累积奖励。 根据以上定义，推荐系统的学习目标就是通过最大化推荐系统的累积奖励找到一个最优的推荐策略 <math id="M25"><msup><mrow><mi>π</mi></mrow><mrow><mi mathvariant="normal">*</mi></mrow></msup></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207021&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207016&type= 3.13266659 3.04799986 ，满足： <math id="M26"><msup><mrow><mi>π</mi></mrow><mrow><mi mathvariant="normal">*</mi></mrow></msup></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207028&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207026&type= 3.13266659 3.04799986 ： <math id="M27"><mi mathvariant="normal">m</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">x</mi><mo stretchy="false">(</mo><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>+</mo><mi>γ</mi><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn mathvariant="normal">1</mn></mrow></msub><mo>+</mo><msup><mrow><mi>γ</mi></mrow><mrow><mn mathvariant="normal">2</mn></mrow></msup><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn mathvariant="normal">2</mn></mrow></msub><mo>+</mo><mo>⋯</mo><mo>+</mo><msup><mrow><mi>γ</mi></mrow><mrow><mi>n</mi><mo>-</mo><mi>t</mi></mrow></msup><msub><mrow><mi>r</mi></mrow><mrow><mi>n</mi></mrow></msub><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207038&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207033&type= 61.04467010 4.91066647 (1) 2　 推荐算法 1 本文融合Item2vec和深度强化学习中的DDPG算法实现电影推荐。DDPG是一种适用于连续动作空间的算法，Item2vec算法可以学习电影项目的词向量表示，将离散的电影动作空间转化为连续的电影动作空间。本文还提出将余弦距离和欧氏距离相结合的奖励函数，帮助算法尽快收敛并获得一个比较好的推荐策略。 2.1　数据处理 2 在强化学习任务中，学习的目标就是找到能够使长期累积奖励最大化的策略 ［ 18 18 ］ 。首先要对历史数据进行处理，将历史评分转换为用户对项目的感 兴趣程度。文献［ 14 14 ］将用户对商品（电商）的忽略、点击、购买行为分别转化为0、1、5来表示用户对该商品的感兴趣程度。这种方法并不适合本文，一是因为本文的数据集中只包含用户的评分值记录；二是这种方法忽略了用户评分值的重要性。使用这种方法，一旦用户对一部电影进行评分，用户对该电影的感兴趣程度就为5，忽略了用户对电影评价的高低。本文认为用户对电影的评分值更能体现推荐是否满足用户兴趣需求，所以感兴趣程度的设置应综合考虑用户的评分值。对每一个用户 <math id="M28"><mi>i</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207049&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207044&type= 1.10066664 2.79399991 的评分数据进行均值计算获得 <math id="M29"><mi mathvariant="normal">u</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">e</mi><msubsup><mrow><mi mathvariant="normal">r</mi></mrow><mrow><mi>r</mi><mo>_</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mrow><mtext> </mtext><mi>i</mi></mrow></msubsup></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207060&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207054&type= 12.44600010 3.97933340 ，对某一电影 <math id="M30"><mi>j</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207067&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207064&type= 1.10066664 3.64066648 在该网站的评分记录进行均值计算获得 <math id="M31"><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>r</mi><mo>_</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mrow><mtext> </mtext><mi>j</mi></mrow></msubsup></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207076&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207072&type= 15.40933418 3.97933340 ，用户 <math id="M32"><mi>i</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207084&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207081&type= 1.10066664 2.79399991 对电影 <math id="M33"><mi>j</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207094&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207088&type= 1.10066664 3.64066648 的评分为 x ，通过 <math display="block" id="M34"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><msub><mrow><mi mathvariant="normal">t</mi></mrow><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow></msub><mo>=</mo><mfenced open="{" close="" separators="|"><mrow><mtable columnalign="left"><mtr><mtd><mn mathvariant="normal">0</mn><mo>,</mo><mi>x</mi><mo><</mo><mi mathvariant="normal">u</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">e</mi><msubsup><mrow><mi mathvariant="normal">r</mi></mrow><mrow><mi>r</mi><mo>_</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mrow><mtext> </mtext><mi>i</mi></mrow></msubsup><mtext> </mtext><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi><mi mathvariant="normal">d</mi><mtext> </mtext><mi>x</mi><mo><</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>r</mi><mo>_</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mrow><mtext> </mtext><mi>j</mi></mrow></msubsup></mtd></mtr><mtr><mtd><mn mathvariant="normal">2</mn><mo>,</mo><mi>x</mi><mo>></mo><mi mathvariant="normal">u</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">e</mi><msubsup><mrow><mi mathvariant="normal">r</mi></mrow><mrow><mi>r</mi><mo>_</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mrow><mtext> </mtext><mi>i</mi></mrow></msubsup><mtext> </mtext><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi><mi mathvariant="normal">d</mi><mtext> </mtext><mi>x</mi><mo>></mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>r</mi><mo>_</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mrow><mtext> </mtext><mi>j</mi></mrow></msubsup></mtd></mtr><mtr><mtd><mn mathvariant="normal">1</mn><mo>,</mo><mi mathvariant="normal">o</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">h</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">w</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">e</mi></mtd></mtr></mtable></mrow></mfenced></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207101&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207097&type= 67.90267181 14.81666660 对数据集的评分数据进行处理， <math id="M37"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207109&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207105&type= 6.26533318 3.30199981 <math id="M38"><msub><mrow/><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207117&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207112&type= 2.28600001 4.06400013 表示用户对该电影的感兴趣程度。 图2 图2 为处理前后的数据。 2.2　连续动作空间表示 2 Barkan等 ［ 17 17 ］ 将Word2vec ［ 19 19 ］ 方法的Skip-gram with negative sampling （SGNS）的思路迁移到基于物品的协同过滤（Item-based CF）上，提出了Item2vec。一个物品集合被看作自然语言中的一个句子，物品集合中的物品等价于句子中的单词，一个商品集合是一个订单中包含的所有商品。以物品的共现性作为自然语言中的上下文关系，构建神经网络来学习物品在空间中的词向量表示。本文将所有用户的历史评分电影记录看作一个语料库，一个用户的评分电影记录看作一个单词集合，通过所有用户的评分记录集合学习每部电影的词向量表示，将离散的电影空间转换为连续的表示。对于用户的评分电影集合 <math id="M39"><msub><mrow><mi>W</mi></mrow><mrow><mn mathvariant="normal">1</mn></mrow></msub><mo>,</mo><msub><mrow><mi>W</mi></mrow><mrow><mn mathvariant="normal">2</mn></mrow></msub><mo>,</mo><mo>⋯</mo><mo>,</mo><msub><mrow><mi>W</mi></mrow><mrow><mi>k</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207152&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207148&type= 24.46866608 4.14866638 ，目标函数为： <math display="block" id="M40"><mi>L</mi><mo>=</mo><mfrac><mrow><mn mathvariant="normal">1</mn></mrow><mrow><mi>K</mi></mrow></mfrac><mstyle displaystyle="true"><munderover><mo largeop="true">∑</mo><mrow><mi>i</mi><mo>=</mo><mn mathvariant="normal">1</mn></mrow><mrow><mi>K</mi></mrow></munderover></mstyle><mrow><mstyle displaystyle="true"><munderover><mo largeop="true">∑</mo><mrow><mi>j</mi><mo>≠</mo><mi>i</mi></mrow><mrow><mi>K</mi></mrow></munderover></mstyle><mrow><mi mathvariant="normal">l</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">g</mi><mi>P</mi><mo stretchy="false">(</mo><msub><mrow><mi>W</mi></mrow><mrow><mi>i</mi></mrow></msub><mfenced open="|" close="" separators="|"><mrow><msub><mrow><mi>W</mi></mrow><mrow><mi>j</mi></mrow></msub><mo stretchy="false">)</mo></mrow></mfenced></mrow></mrow></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207160&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207156&type= 42.41799927 8.88999939 <math display="block" id="M41"><mi>P</mi><mo stretchy="false">(</mo><msub><mrow><mi>W</mi></mrow><mrow><mi>i</mi></mrow></msub><mfenced open="|" close="" separators="|"><mrow><msub><mrow><mi>W</mi></mrow><mrow><mi>j</mi></mrow></msub><mo stretchy="false">)</mo><mo>=</mo><mi>σ</mi><mo stretchy="false">(</mo></mrow></mfenced><msubsup><mrow><mi>u</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="normal">T</mi></mrow></msubsup><msub><mrow><mi>v</mi></mrow><mrow><mi>j</mi></mrow></msub><mo stretchy="false">)</mo><mstyle displaystyle="true"><munderover><mo>∏</mo><mrow><mi>k</mi><mo>=</mo><mn mathvariant="normal">1</mn></mrow><mrow><mi>N</mi></mrow></munderover></mstyle><mrow><mi>σ</mi><mo stretchy="false">(</mo><mo>-</mo><msubsup><mrow><mi>u</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="normal">T</mi></mrow></msubsup><msub><mrow><mi>v</mi></mrow><mrow><mi>k</mi></mrow></msub><mo stretchy="false">)</mo></mrow></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207169&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207164&type= 54.18666840 8.38199997 其中， <math id="M42"><mi>σ</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mn mathvariant="normal">1</mn><mo>/</mo><mo stretchy="false">(</mo><mn mathvariant="normal">1</mn><mo>+</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">x</mi><mi mathvariant="normal">p</mi><mo stretchy="false">(</mo><mo>-</mo><mi>x</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207177&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207173&type= 40.38600159 3.89466667 ， u <math id="M43"><mo>∈</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207184&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207180&type= 2.70933342 3.13266659 W <math id="M44"><msub><mrow/><mrow><mi>i</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207189&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207186&type= 0.84666669 4.06400013 ， v <math id="M45"><mo>∈</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207195&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207192&type= 2.70933342 3.13266659 W <math id="M46"><msub><mrow/><mrow><mi>j</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207201&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207198&type= 0.76199996 4.06400013 ， N 为对于每一个正样本负采样的个数。 2.3　奖励函数设置 2 Algorithm 1 Recommender Simulator Input： User rating history The length of state K Output： Memory M for user=1… n do for i in len(user rating list) do <math id="M47"><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> ={ <math id="M48"><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msub><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>i</mi></mrow></msub><mo>,</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msub><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>i</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>,</mo><mo>…</mo><mo>,</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msub><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>i</mi><mo>+</mo><mi>K</mi><mo>-</mo><mn>1</mn></mrow></msub></math> } a <math id="M49"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> = <math id="M50"><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msub><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>i</mi><mo>+</mo><mi>K</mi></mrow></msub></math> <math id="M51"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> = r _ht of <math id="M52"><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msub><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>i</mi><mo>+</mo><mi>K</mi></mrow></msub></math> if <math id="M53"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> > 0 then Add <math id="M54"><mo stretchy="false">(</mo><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> ， a <math id="M55"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> ， <math id="M56"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> ) in M end if end for end for Return M 奖励函数在强化学习中至关重要，奖励函数影响算法的收敛速度和学习效果。本文对用户的历史数据进行处理，将用户的历史评分列表转换为（ <math id="M57"><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207214&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207211&type= 6.01133347 3.30199981 ， a <math id="M58"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207221&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207218&type= 4.82599974 3.30199981 ， <math id="M59"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207227&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207224&type= 6.26533318 3.30199981 ）的形式，并将其存储在 M 中（ 算法1 算法1 ）。推荐Agent在 t 时刻，从 M 获得一个（ <math id="M60"><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207235&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207233&type= 6.01133347 3.30199981 ， a <math id="M61"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207240&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207237&type= 4.82599974 3.30199981 ， <math id="M62"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207248&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207244&type= 6.26533318 3.30199981 ）元组，令状态 <math id="M63"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>=</mo><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo>=</mo><mfenced open="{" close="" separators="|"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>t</mi></mrow><mrow><mn mathvariant="normal">1</mn></mrow></msubsup></mrow></mfenced><mo>,</mo><mo>…</mo><mfenced open="" close="}" separators="|"><mrow><mo>,</mo><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">v</mi><mi mathvariant="normal">i</mi><msubsup><mrow><mi mathvariant="normal">e</mi></mrow><mrow><mi>t</mi></mrow><mrow><mi>K</mi></mrow></msubsup></mrow></mfenced></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207254&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207251&type= 50.71533203 4.48733330 ，产生一个电影推荐 <math id="M64"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207259&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207256&type= 2.62466669 3.72533321 ，通过原数据中的 a <math id="M65"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207264&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207262&type= 4.82599974 3.30199981 和 r <math id="M66"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207270&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207268&type= 4.82599974 3.30199981 对该电影推荐进行评估并给予一个立即奖励 <math id="M67"><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207279&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207275&type= 2.28600001 3.72533321 （ <math id="M68"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207286&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207282&type= 2.03200006 3.72533321 ， <math id="M69"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207292&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207290&type= 2.62466669 3.72533321 ）。现有的研究通常利用两者之间的余弦距离计算奖励值，但实验发现余弦距离更倾向于从方向上区分差异，对距离并不敏感，很容易使算法过早收敛于一个局部最优的策略，产生余弦相似度较高，但欧氏距离很大的推荐结果，无法满足用户的要求。本文对用户短时期内评分的电影进行相似度分析（如 表1 表1 所示，表中 <math id="M70"><msub><mrow><mi>m</mi></mrow><mrow><mi>i</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207300&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207295&type= 3.55599999 3.72533321 表示电影ID），可以看出用户的电影兴趣在短期内没有太大变化，不同电影间余弦相似度比较高（ <math id="M71"><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">n</mi><mo>></mo><mn mathvariant="normal">0</mn></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207306&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207303&type= 14.39333344 2.79399991 ）且欧氏距离比较小（ <math id="M72"><mi mathvariant="normal">E</mi><mi mathvariant="normal">d</mi><mo><</mo><mn mathvariant="normal">1</mn></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207314&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207310&type= 11.09133339 2.79399991 ），因此本文设计了一种余弦相似度和欧氏距离相结合的奖励函数，如下式： <math display="block" id="M73"><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo>=</mo><mfenced open="{" close="" separators="|"><mrow><mtable columnalign="left"><mtr><mtd><mn mathvariant="normal">10</mn><mo>,</mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mo>≥</mo><mn mathvariant="normal">0.5</mn><mo>,</mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi><mo>≤</mo><mn mathvariant="normal">0.5</mn></mtd></mtr><mtr><mtd><mn mathvariant="normal">5,0.5</mn><mo>></mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mo>≥</mo><mn mathvariant="normal">0.3</mn><mo>,</mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi><mo>+</mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mo>≤</mo><mn mathvariant="normal">1</mn></mtd></mtr><mtr><mtd><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">*</mi><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo>+</mo><mo stretchy="false">(</mo><mi mathvariant="normal">e</mi><msub><mrow><mi mathvariant="normal">d</mi></mrow><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">x</mi></mrow></msub><mo>-</mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi><mo stretchy="false">)</mo><mi mathvariant="normal">*</mi><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo>,</mo><mi mathvariant="normal">o</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">h</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">w</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">e</mi></mtd></mtr></mtable></mrow></mfenced></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207320&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207317&type= 93.81066895 15.74800014 <math display="block" id="M74"><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mo stretchy="false">(</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><mi>a</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo stretchy="false">)</mo><mo>=</mo><mo stretchy="false">(</mo><msub><mrow><mi mathvariant="bold-italic">a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>⋅</mo><mi mathvariant="bold-italic">a</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo stretchy="false">)</mo><mo>/</mo><mo stretchy="false">(</mo><mo stretchy="false">|</mo><mo stretchy="false">|</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">|</mo><mo stretchy="false">|</mo><mo>⋅</mo><mo stretchy="false">|</mo><mo stretchy="false">|</mo><mi>a</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo stretchy="false">|</mo><mo stretchy="false">|</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207325&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207323&type= 66.46333313 4.57200003 ） <math display="block" id="M75"><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi><mo stretchy="false">(</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><mi>a</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo stretchy="false">)</mo><mo>=</mo><mroot><mrow><mo>∑</mo><mo stretchy="false">(</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><msup><mrow><mo>-</mo><mi>a</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo stretchy="false">)</mo></mrow><mrow><mn mathvariant="normal">2</mn></mrow></msup></mrow><mrow/></mroot></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207331&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207328&type= 56.38800049 6.01133347 其中， <math id="M76"><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">c</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207337&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207334&type= 12.10733318 3.30199981 是用户实际观看电影 <math id="M77"><mi>a</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207343&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207340&type= 6.68866634 3.30199981 和推荐电影 <math id="M78"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207348&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207346&type= 2.62466669 3.72533321 间的余弦距离， <math id="M79"><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207353&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207350&type= 10.66799927 3.30199981 是两部电影间的欧氏距离， <math id="M80"><mi mathvariant="normal">e</mi><msub><mrow><mi mathvariant="normal">d</mi></mrow><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">x</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207357&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207355&type= 7.11199999 3.72533321 是欧氏距离的惩罚因子，当 <math id="M81"><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi><mo>></mo><mi mathvariant="normal">e</mi><msub><mrow><mi mathvariant="normal">d</mi></mrow><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">x</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207362&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207360&type= 22.60599899 4.23333359 时， <math id="M82"><mi mathvariant="normal">e</mi><msub><mrow><mi mathvariant="normal">d</mi></mrow><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">x</mi></mrow></msub><mo>-</mo><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">m</mi><mo>_</mo><mi mathvariant="normal">e</mi><mi mathvariant="normal">d</mi><mo><</mo><mn mathvariant="normal">0</mn></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207365&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207363&type= 29.29466629 4.23333359 会使reward值降低。 2 <math id="M83"><mo stretchy="false">(</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>2</mn></mrow></msub><mo>,</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>3</mn></mrow></msub><mo>,</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>4</mn></mrow></msub><mo stretchy="false">)</mo></math> 2 <math id="M84"><mo stretchy="false">(</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>4</mn></mrow></msub><mo stretchy="false">)</mo></math> 2 <math id="M85"><mo stretchy="false">(</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>2</mn></mrow></msub><mo>,</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>4</mn></mrow></msub><mo stretchy="false">)</mo></math> 2 <math id="M86"><mo stretchy="false">(</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>3</mn></mrow></msub><mo>,</mo><msub><mrow><mi>m</mi></mrow><mrow><mn>4</mn></mrow></msub><mo stretchy="false">)</mo></math> cosin Ed cosin Ed cosin Ed (31,1 029,1 061, 1 129) 0.42 0.85 0.71 0.51 0.46 0.67 (1 029,1 061,1 129,1 263) 0.75 0.42 0.63 0.50 0.81 0.44 (5 218,5 299,5 377,5 445) 0.80 0.36 0.82 0.35 0.67 0.49 2.4　改进的强化学习算法DDPG 2 传统的强化学习算法常应用于在线游戏的训练，在这种情况下Agent面对任何状态都可以采取相应动作并即时获得奖励和状态的更新。对于推荐系统来说，在算法上线应用之前是无法获得用户真正的反馈的，只能通过模拟的方法来获得每一次推荐带来的奖励。本文通过 M 和设置的奖励函数获得推荐的模拟奖励值，提出一种改进的DDPG算法。传统DDPG算法在每个episode之前初始化状态，本文令算法在每一个时间步都从推荐模拟存储器 M 获得一个已知状态（见 算法2 算法2 ），这样Agent产生的推荐的奖励值就得以计算。令 <math id="M87"><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mfenced open="" close="|" separators="|"><mrow><mi>a</mi></mrow></mfenced><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207375&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207372&type= 14.90133286 3.81000018 表示Critic网络， <math id="M88"><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207380&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207378&type= 3.04799986 3.04799986 为神经网络参数，用于对Actor产生的推荐进行评价， <math id="M89"><msub><mrow><mi>f</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207384&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207382&type= 3.04799986 4.23333359 为Actor网络用来产生推荐， <math id="M90"><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207390&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207387&type= 2.96333337 3.04799986 为其对应的神经网络参数。在训练过程中，对于当前状态 <math id="M91"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207394&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207392&type= 2.03200006 3.72533321 ，推荐Agent产生一个电影推荐（动作） <math id="M92"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207398&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207396&type= 2.62466669 3.72533321 ，用户通过（5）式计算立即奖励 <math id="M93"><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207400&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207399&type= 2.28600001 3.72533321 （ s t ， a t ），同时更新状态 <math id="M94"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn mathvariant="normal">1</mn></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207402&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207401&type= 5.67266655 3.72533321 。将数据以 <math id="M95"><mo stretchy="false">(</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn mathvariant="normal">1</mn></mrow></msub></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207404&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207403&type= 19.89666748 4.48733330 ）的形式存储在存储器 D 中，算法每次从 D 中随机采样 m 个元组进行学习。 Critic网络对状态-动作对的价值函数 <math id="M96"><mi>Q</mi><mo stretchy="false">(</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207406&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207405&type= 12.78466702 4.48733330 进行学习，Critic的损失函数为： <math display="block" id="M97"><mi>y</mi><mo>=</mo><mi>r</mi><mo>+</mo><mi>γ</mi><mi>Q</mi><mi mathvariant="normal">'</mi><mo stretchy="false">(</mo><mi>s</mi><mi mathvariant="normal">'</mi><mo>,</mo><msub><mrow><mi>f</mi><mi>'</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi><mi mathvariant="normal">'</mi></mrow></msup></mrow></msub><mo stretchy="false">|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi><mi>'</mi></mrow></msup><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207408&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207407&type= 34.71333313 5.07999992 <math display="block" id="M98"><mi mathvariant="normal">L</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">s</mi><mo>=</mo><mfrac><mrow><mn mathvariant="normal">1</mn></mrow><mrow><mi>m</mi></mrow></mfrac><mstyle displaystyle="true"><munderover><mo largeop="true">∑</mo><mrow><mi>t</mi><mo>=</mo><mn mathvariant="normal">1</mn></mrow><mrow><mi>m</mi></mrow></munderover></mstyle><mrow><mo stretchy="false">(</mo><mi>y</mi><mo>-</mo><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mi>a</mi><mo stretchy="false">|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><msup><mrow><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mrow><mn mathvariant="normal">2</mn></mrow></msup></mrow></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207410&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207409&type= 47.66733170 8.38199997 其中， y 为Critic-target网络的值， <math id="M99"><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mi>a</mi><mo stretchy="false">|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207413&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207412&type= 14.90133286 3.81000018 是Critic网络的值， <math id="M100"><mi>γ</mi><mo>∈</mo><mo stretchy="false">[</mo><mn mathvariant="normal">0,1</mn><mo stretchy="false">]</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207416&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207414&type= 14.64733315 3.72533321 是折扣因子，然后使用梯度下降法对网络进行更新： <math display="block" id="M101"><msub><mrow><mo>∇</mo></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup></mrow></msub><mi>L</mi><mo stretchy="false">(</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn mathvariant="normal">1</mn></mrow><mrow><mi>m</mi></mrow></mfrac><mo stretchy="false">[</mo><mo stretchy="false">(</mo><mi>y</mi><mo>-</mo><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mi>a</mi><mo stretchy="false">|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo><msub><mrow><mo>∇</mo></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup></mrow></msub><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mi>a</mi><mo stretchy="false">|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo><mo stretchy="false">]</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207419&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207418&type= 71.28932953 7.53533316 在Actor-Critic框架中，Actor根据当前状态选择动作，Critic评价该动作的得分，Actor再根据Critic的评分修改该动作被选择的概率。Actor是一种基于策略的学习网络，用策略函数 J 来评价一个策略的好坏，对策略函数求梯度： <math display="block" id="M102"><msub><mrow><mo>∇</mo></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub><msub><mrow><mi>J</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub><mo>=</mo><mfrac><mrow><mn mathvariant="normal">1</mn></mrow><mrow><mi>m</mi></mrow></mfrac><munder><mo movablelimits="false">∑</mo><mrow><mi>i</mi></mrow></munder><mrow><msub><mrow><mo>∇</mo></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup></mrow></msub><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mi>a</mi><mo stretchy="false">|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo></mrow><msub><mrow><mo>∇</mo></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub><msub><mrow><mi>f</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></math> http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207422&type= http://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=4207420&type= 57.91199875 8.38199997 Algorithm 2 改进的DDPG算法 Random initialize Critic network <math id="M103"><mi>Q</mi><mo stretchy="false">(</mo><mi>s</mi><mo>,</mo><mi>a</mi><mo>|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo stretchy="false">)</mo></math> and Actor <math id="M104"><msub><mrow><mi>f</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub></math> with weights Initialize target network <math id="M105"><mi>Q</mi><mi>'</mi></math> and <math id="M106"><mi>f</mi><mtext> </mtext><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mi mathvariant="normal">'</mi></math> with weights <math id="M107"><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi><mi>'</mi></mrow></msup><mo>←</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo>,</mo><msup><mrow><mi mathvariant="normal">θ</mi></mrow><mrow><mi>ρ</mi><mi>'</mi></mrow></msup><mo>←</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></math> Initialize the capacity of replay memory D Initialize a Gaussian Noise G for episode = 1… N do for t = 1… T do Select a tuple <math id="M108"><mo stretchy="false">(</mo><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> ， a <math id="M109"><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> ， <math id="M110"><mi>r</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi><mo stretchy="false">)</mo></math> from R and <math id="M111"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub></math> = <math id="M112"><mi>s</mi><mo>_</mo><mi mathvariant="normal">h</mi><mi mathvariant="normal">t</mi></math> <math id="M113"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> = <math id="M114"><msub><mrow><mi>f</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup></mrow></msub><mo stretchy="false">(</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo>+</mo><mi>G</mi></math> according to current policy and exploration noise execute <math id="M115"><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub></math> and observe <math id="M116"><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub></math> according equation (5) equation (5) observe new state <math id="M117"><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub></math> Store transition <math id="M118"><mo stretchy="false">(</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>a</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>r</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>,</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></math> in D Sample mini⁃batch of m transitions ( s , a , r , s ′) from D Set <math id="M119"><mi>y</mi><mo>=</mo><mi>r</mi><mo>+</mo><mi>γ</mi><mi>Q</mi><mi mathvariant="normal">'</mi><mo stretchy="false">(</mo><mi>s</mi><mi mathvariant="normal">'</mi><mo>,</mo><mi>f</mi><mtext> </mtext><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><mo> </mo><msub><mrow><mi mathvariant="normal">'</mi></mrow><mrow><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi><mi>'</mi></mrow></msup></mrow></msub><mo>|</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi><mi>'</mi></mrow></msup><mo stretchy="false">)</mo></math> Update Critic by minimizing the loss : equation (10) equation (10) Update the Actor using the sampled policy gradient: equation (11) equation (11) Update the target networks <math id="M120"><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi><mi>'</mi></mrow></msup><mo>←</mo><mi>τ</mi><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi></mrow></msup><mo>+</mo><mo stretchy="false">(</mo><mn>1</mn><mo>-</mo><mi>τ</mi><mo stretchy="false">)</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>μ</mi><mi>'</mi></mrow></msup><mo>,</mo></math> <math id="M121"><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi><mi>'</mi></mrow></msup><mo>←</mo><mi>τ</mi><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi></mrow></msup><mo>+</mo><mo stretchy="false">(</mo><mn>1</mn><mo>-</mo><mi>τ</mi><mo stretchy="false">)</mo><msup><mrow><mi>θ</mi></mrow><mrow><mi>ρ</mi><mi>'</mi></mrow></msup></math> end for end for 3　 实 验 1 实验硬件环境：PC机一台，8GB内存+256GB硬盘。本文实验平台操作系统为window10，编程语言为python语言，编程在Jupyter Notebook上进行。实验使用movielens数据集（ https：//grouplens.org/datasets/movielens/ https://grouplens.org/datasets/movielens/ ），其中ml-latest-small数据集中包括600个用户对9 000部电影的10万条评分记录。设 K =3（状态中包括3部电影），实验在经过Item2vec处理后的连续动作空间上进行，电影的词向量维度分别设置为20，50，70，对比不同词向量维度在相同算法参数设置下，平均奖励的收敛情况和推荐准确性。 图3 图3 为在不同词向量维度设置下平均奖励的收敛情况，可以看出该算法在词向量维度为20时，产生的推荐策略获得的平均奖励最大，推荐的电影最符合用户兴趣。本实验选取准确率（Precision）作为推荐策略的评价指标， 表2 表2 为算法在不同词向量维度设置下的推荐准确度。 词向量维数 准确率@5 准确率@10 20 0.271 0.304 50 0.249 0.255 70 0.235 0.240 冷启动问题是指，推荐系统在用户数据比较少的情况下难以学习用户偏好、产生令用户满意的推荐。实验同时在ml-20m的数据集上进行训练，该数据集包括13万用户对27 000部电影的2×10 7 条评分记录，对比算法在两个数据集上的推荐准确率（ 表3 表3 ）可以发现，ml-20m数据集的评论数是ml-latest-small的2.0×10 2 倍，但两者的推荐准确率@10只有0.007的差距，所以本文提出的算法在一定程度上能够缓解推荐系统中存在的冷启动问题。 数据集 评论数 准确率@10 ml-latest-small 1.0×10 5 0.304 ml-20m 2.0×10 7 0.311 4　 结 语 1 本文提出一种Item2vec和改进的DDPG相结合的推荐算法。使用Item2vec将离散的电影项目转化为词向量表示，同时提出了一种余弦距离与欧氏距离相结合的奖励函数，实现了较好的推荐准确度。实验在两个大小不同的数据集上进行，验证了本文提出的算法在缓解冷启动问题上也有很好的表现。本文提出的算法还可以进行扩展，对于状态的设置增加其他因素，如时间、用户当前的搜索关键字等，以进一步提高推荐准确性。 参考文献 PETER B ， ALFRED K ， WOLFGANG N . An Adaptive Web ［M］. Berlin ： Springer ， 2007 ： 291 - 324 . An Adaptive Web RICCI F ， ROKACH L ， SHAPIRA B ， et al . Recommender Systems Handbook ［M］. Boston ： Springer ， 2011 ： 145 - 186 . Recommender Systems Handbook OKURA S ， TAGAMI Y ， ONO S ， et al . Embedding-based news recommendation for millions of users ［C］// Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . New York ： ACM ， 2017 ： 1933 - 1942 . DOI： 10.1145/3097983.3098108 http://dx.doi.org/10.1145/3097983.3098108 . Embedding-based news recommendation for millions of users COVINGTON P ， ADAMS J ， SARGIN E . Deep neural networks for YouTube recommendations ［C］// ACM Conference on Recommender Systems . New York ： ACM ， 2016 ： 191 - 198 . Deep neural networks for YouTube recommendations AMATRIAIN X . The recommender problem revisited ［C］// Proceedings of the 8th ACM Conference on Recommender Systems . New York ： ACM ， 2014 ： 397 - 398 . The recommender problem revisited SUTTON R ， BARTO A . Reinforcement learning：An introduction ［J］. IEEE Transactions on Neural Networks ， 1998 ， 9 （ 5 ）： 1054 - 1054 . DOI： 10.1109/tnn.1998.712192 http://dx.doi.org/10.1109/tnn.1998.712192 . Reinforcement learning：An introduction SHANI G ， HECKERMAN D ， BRAFMAN R I . An MDP-based recommender system ［J］. Journal of Machine Learning Research ， 2005 ， 6 （ 1 ）： 1265 - 1295 . DOI： 10.1007/s10846-005-9023-3 http://dx.doi.org/10.1007/s10846-005-9023-3 . An MDP-based recommender system TAGHIPOUR N ， KARDAN A . A hybrid web recommender system based on Q-learning ［C］// ACM Symposium on Applied Computing . New York ： ACM ， 2008 ： 1164 - 1168 . DOI： 10.1109/GrC.2010.153 http://dx.doi.org/10.1109/GrC.2010.153 . A hybrid web recommender system based on Q-learning CHOI S ， HA H ， HWANG U ， et al . Reinforcement learning based recommender system using bi-clustering technique ［DB/OL］.［ 2018-10-02 ］. https：//arxiv.org/pdf/1801.05532.pdf https://arxiv.org/pdf/1801.05532.pdf . Reinforcement learning based recommender system using bi-clustering technique MNIH V ， KAVUKCUGLU K ， SILVER D ， et al . Playing Atari with deep reinforcement learning ［DB/OL］.［ 2018-10-02 ］. https：//arxiv.org/pdf/1312.5602v1.pdf https://arxiv.org/pdf/1312.5602v1.pdf . Playing Atari with deep reinforcement learning MNIH V ， KAVUKCUGLU K ， SILVER D ， et al . Human-level control through deep reinforcement learning ［J］. Nature ， 2015 ， 518 （ 7540 ）： 529 - 533 . DOI： 10.1038/nature14236 http://dx.doi.org/10.1038/nature14236 . Human-level control through deep reinforcement learning SILVER D ， LEVER G ， HEESS N ， et al . Deterministic Policy Gradient Algorithms ［DB/OL］.［ 2018-12-12 ］. http：//www0.cs.ucl.ac.uk/staff/d.silver/web/Publications_files/deterministic-policy-gradients.pdf http://www0.cs.ucl.ac.uk/staff/d.silver/web/Publications_files/deterministic-policy-gradients.pdf . Deterministic Policy Gradient Algorithms LILLICRAP T P ， HUNT J J ， PRITZEL A ， et al . Continuous control with deep reinforcement learning ［J］. Computer Science ， 2015 ， 6 ： A187 . Continuous control with deep reinforcement learning ZHAO X Y ， ZHANG L ， DING Z ， et al . Deep reinforcement learning for list-wise recommendations ［J］. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York：ACM ， 2018 ： 95 - 103 . DOI： 10.1145/3240323.3240374 http://dx.doi.org/10.1145/3240323.3240374 . Deep reinforcement learning for list-wise recommendations ZHAO X Y ， ZHANG L ， DING Z ， et al . Recommendations with negative feedback via pairwise deep reinforcement learning ［C］// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . New York ： ACM ， 2018 ： 1040 - 1048 ， DOI： 10.1145/3219819.3219886 http://dx.doi.org/10.1145/3219819.3219886 . Recommendations with negative feedback via pairwise deep reinforcement learning MUNEMASA I ， TOMOMATSU Y ， HAYASHI K ， et al . Deep reinforcement learning for recommender systems ［C］ // International Conference on Information & Communications Technology . New York ： IEEE ， 2018 ： 226 - 233 . Deep reinforcement learning for recommender systems BARKAN O ， KOENIGSTEIN N . Item2Vec： Neural Item Embedding for Collaborative Filtering ［C］// 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing （MLSP） . New York ： IEEE ， 2016 ： 1 - 6 . DOI： 10.1109/MLSP.2016.7738886 http://dx.doi.org/10.1109/MLSP.2016.7738886 . Item2Vec： Neural Item Embedding for Collaborative Filtering 周志华 . 机器学习 ［M］. 北京 ： 清华大学出版社 ， 2016 ： 371 - 397 . 机器学习 ZHOU Z H . Machine Learning ［M］. Beijing ： Tsinghua University Press ， 2016 ： 371-397（Ch） . 371-397（Ch） MIKOLOV T ， CHEN K ， CORRADO G ， et al . Efficient Estimation of Word Representations in Vector Space ［DB/OL］. ［ 2018-12-20 ］. https：//www.researchgate.net/publication/234131319_Efficient_Estimation_of_Word_Representations_in_Vector_Space https://www.researchgate.net/publication/234131319_Efficient_Estimation_of_Word_Representations_in_Vector_Space . Efficient Estimation of Word Representations in Vector Space 10.14188/j.1671-8836.2019.03.010 TP 301.6 A 1671-8836（2019）03-0297-06 LIU Wenzhu,HUANG Bo,GAO Yongbin, et al .Recommendation Algorithm Combining Item2vec with Improved DDPG [J]. J Wuhan Univ （ Nat Sci Ed ）,2019,65(3):297-302. DOI: 10.14188/j.1671-8836.2019.03.010 http://dx.doi.org/10.14188/j.1671-8836.2019.03.010 (Ch). LIU Wenzhu,HUANG Bo,GAO Yongbin,et al.Recommendation Algorithm Combining Item2vec with Improved DDPG [J].J Wuhan Univ（Nat Sci Ed）,2019,65(3):297-302. DOI:10.14188/j.1671-8836.2019.03.010(Ch). 刘文竹,黄勃,高永彬, 等.Item2vec与改进DDPG相融合的推荐算法[J].武汉大学学报（理学版）,2019,65(3):297-302. DOI:10.14188/j.1671-8836.2019.03.010 刘文竹,黄勃,高永彬, 等.Item2vec与改进DDPG相融合的推荐算法[J].武汉大学学报（理学版）,2019,65(3):297-302. DOI:10.14188/j.1671-8836.2019.03.010 国家自然科学基金青年基金（61603242）；江西省经济犯罪侦查与防控技术协同创新中心开放基金（JXJZXTCX-030） 2018-07-08 2019-06-24 2020-11-28 武汉大学学报（理学版） 3 65