KL divergence (Kullback-Leibler divergence)\u7528\u6765\u8861\u91cf\u4e24\u4e2a\u5206\u5e03\u7684\u5dee\u5f02\uff0c\u4e00\u822c\u6807\u8bb0\u4e3a$D_{KL}(P||Q)$\uff0c\u79bb\u6563\u5f62\u5f0f\u7684\u8ba1\u7b97\u516c\u5f0f\u662f<\/p>\n\n\n\n
$$D_{KL}(P||Q) = \\sum_{x\\in\\chi}P(x) \\log \\frac{P(x)}{Q(x)}$$<\/p>\n\n\n\n
\u4ece\u516c\u5f0f\u4e0a\u770b\uff0c\u5b83\u8ba1\u7b97\u7684\u662f$P(x)$\u548c$Q(x)$\u7684log\u5dee\u7684\u5747\u503c\uff0c$x$\u670d\u4ece$P(x)$\u5206\u5e03\uff0c\u56e0\u4e3a$\\log \\frac{P(x)}{Q(x)} = \\log P(x) – \\log Q(x)$\u3002<\/p>\n\n\n\n
\u5728\u901a\u4fe1\u9886\u57df\u5b83\u53eb\u76f8\u5bf9\u71b5(relative entropy)\uff0c\u5728\u673a\u5668\u5b66\u4e60\u4efb\u52a1\u4e2d\u53ef\u4ee5\u901a\u8fc7\u6700\u5c0f\u5316KL divergence\u6765\u5b66\u4e60\u76ee\u6807\u5206\u5e03$P(x)$\uff0c\u4e0d\u591f\u6211\u4eec\u66f4\u5e38\u7528\u7684\u662f\u4ea4\u53c9\u71b5(Cross Entropy)\uff0c\u628aKL divergence\u516c\u5f0f\u5c55\u5f00\u5c31\u53ef\u4ee5\u5f97\u5230\u4ea4\u53c9\u71b5\u7684\u8ba1\u7b97\u516c\u5f0f<\/p>\n\n\n\n
$$D_{KL}(P||Q) = \\sum_{x\\in\\chi}P(x) \\log \\frac{P(x)}{Q(x)} = \\sum_{x\\in\\chi}P(x) \\left [ \\log P(x) – \\log Q(x) \\right ] = \\sum_{x\\in\\chi}P(x) \\log P(x) – \\sum_{x\\in\\chi}P(x) \\log Q(x)$$<\/p>\n\n\n\n
\u76ee\u6807\u662f$P(x)$\u662f\u5df2\u77e5\u7684\uff0c\u6211\u4eec\u901a\u8fc7\u8c03\u6574\u6a21\u578b$Q(x)$\u7684\u53c2\u6570\u6700\u5c0f\u5316KL divergence\u4ece\u800c\u903c\u8fd1$P(x)$\uff0c\u516c\u5f0f\u5de6\u8fb9\u7684$\\sum_{x\\in\\chi}P(x) \\log P(x)$\u662f\u4e00\u4e2a\u5e38\u6570\uff0c\u4e8e\u662f\u5269\u4e0b\u7684\u8fd9\u90e8\u5206$-\\sum_{x\\in\\chi}P(x) \\log Q(x)$\u662f\u5b9e\u9645\u6709\u7528\u7684\u90e8\u5206\uff0c\u8fd9\u90e8\u5206\u5c31\u662f\u673a\u5668\u5b66\u4e60\u4e2d\u7ecf\u5e38\u4f7f\u7528\u7684\u4ea4\u53c9\u71b5\u635f\u5931\u51fd\u6570\uff0c\u6bd4\u5982\u4e8c\u4efd\u7c7b\u95ee\u9898\u7684\u8bdd<\/p>\n\n\n\n
$$\\text{Binary Cross Entropy} = \\sum_{x\\in\\chi} \\left [ y \\log \\hat{y} \\right ] = \\sum_{x\\in\\chi} \\left [ y\\cdot \\log \\hat{y} + (1-y) \\cdot \\log (1-\\hat{y}) \\right ]$$<\/p>\n\n\n\n
\u518d\u8fdb\u4e00\u6b65\u5206\u6790\u4e0b\uff0c\u5728\u4e8c\u5206\u7c7b\u4efb\u52a1\u4e2d\u6211\u4eec\u4e5f\u53ef\u4ee5\u4f7f\u7528\u6700\u5927\u4f3c\u7136\u4f30\u8ba1\u6765\u83b7\u5f97\u6a21\u578b\u7684\u53c2\u6570\uff0c\u8fd9\u91cc\u76ee\u6807label\u53d8\u6210\u6837\u672c\u662f\u6b63\u6837\u672c\u7684\u6982\u7387\uff0c\u6240\u4ee5\u6b63\u6837\u672c\u7684\u6982\u7387\u662f1\uff0c\u8d1f\u6837\u672c\u7684\u6982\u7387\u662f0\u3002<\/p>\n\n\n\n
$$L(\\theta) = \\prod_{i=1}^{N} Q(x_i|\\theta)$$<\/p>\n\n\n\n
\u6700\u5927\u5316\u4e0a\u8ff0\u516c\u5f0f\u540c\u6837\u53ef\u4ee5\u89e3\u51b3\u6211\u4eec\u7684\u5206\u7c7b\u4efb\u52a1\uff0c\u7b97\u4e58\u6cd5\u4e0d\u5982\u7b97\u52a0\u6cd5\u65b9\u4fbf\uff0c\u800c\u4e14$Q(x)$\u8f93\u51fa\u7684\u503c\u672c\u6765\u5c31\u5c0f\uff0c\u8fde\u4e58\u4ee5\u540e\u5c31\u66f4\u5c0f\u4e86\uff0c\u6240\u4ee5\u8fd9\u91cc\u52a0\u4e2a$\\log$\u53d8\u6210:<\/p>\n\n\n\n
$$\\log L(\\theta) = \\log \\prod_{i=1}^{N} Q(x_i|\\theta) = \\sum_{i=1}^{N} \\log Q(x_i|\\theta)$$<\/p>\n\n\n\n
$Q(x_i)$\u5373\u9884\u6d4b\u503c\uff0c\u5199\u6210$\\hat{y}_i$\uff0c\u6700\u5927\u4f3c\u7136\u4f30\u8ba1\u7684\u76ee\u6807\u5373$\\sum_{i=1}^{N} \\log \\hat{y}_i$\uff0c\u672c\u8eab\u9690\u542b\u7684\u5c31\u662f$\\sum_{i=1}^{N} y_i \\log \\hat{y}_i$\uff0c\u52a0\u4e0a\u8d1f\u53f7\u4e5f\u5c31\u53d8\u6210\u4e86\u6700\u5c0f\u5316$-\\sum_{i=1}^{N} y_i \\log \\hat{y}_i$\u3002<\/p>\n\n\n\n
\u4ece\u4e0a\u9762\u7684\u5206\u6790\u770b\uff0c\u6700\u5c0f\u5316\u4ea4\u53c9\u71b5\u635f\u5931\u51fd\u6570\u548c\u6700\u5927\u4f3c\u7136\u4f30\u8ba1\u672c\u8eab\u662f\u60f3\u901a\u7684\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"
KL divergence (Kullback-Leibler divergence)\u7528\u6765\u8861\u91cf\u4e24\u4e2a\u5206\u5e03\u7684\u5dee\u5f02\uff0c\u4e00\u822c\u6807\u8bb0\u4e3a$D_{KL}(P||Q)$\uff0c\u79bb\u6563\u5f62\u5f0f\u7684\u8ba1\u7b97\u516c\u5f0f\u662f $$D_{KL}(P||Q) = \\sum_{x\\in\\chi}P(x) \\log \\frac{P(x)}{Q(x)}$$ \u4ece\u516c\u5f0f\u4e0a\u770b\uff0c\u5b83\u8ba1\u7b97\u7684\u662f$P(x)$\u548c$Q(x)$\u7684log\u5dee\u7684\u5747\u503c\uff0c$x$\u670d\u4ece$P(x)$\u5206\u5e03\uff0c\u56e0\u4e3a$\\log \\frac{P(x)}{Q(x)} = \\log P(x) – \\log Q(x)$\u3002 \u5728\u901a\u4fe1\u9886\u57df\u5b83\u53eb\u76f8\u5bf9\u71b5(relative entropy)\uff0c\u5728\u673a\u5668\u5b66\u4e60\u4efb\u52a1\u4e2d\u53ef\u4ee5\u901a\u8fc7\u6700\u5c0f\u5316KL divergence\u6765\u5b66\u4e60\u76ee\u6807\u5206\u5e03$P(x)$\uff0c\u4e0d\u591f\u6211\u4eec\u66f4\u5e38\u7528\u7684\u662f\u4ea4\u53c9\u71b5(Cross Entropy)\uff0c\u628aKL divergence\u516c\u5f0f\u5c55\u5f00\u5c31\u53ef\u4ee5\u5f97\u5230\u4ea4\u53c9\u71b5\u7684\u8ba1\u7b97\u516c\u5f0f $$D_{KL}(P||Q) = \\sum_{x\\in\\chi}P(x) \\log \\frac{P(x)}{Q(x)} = […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-796","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts\/796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=796"}],"version-history":[{"count":25,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts\/796\/revisions"}],"predecessor-version":[{"id":824,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts\/796\/revisions\/824"}],"wp:attachment":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}