{"id":416,"date":"2021-04-01T12:43:00","date_gmt":"2021-04-01T12:43:00","guid":{"rendered":"https:\/\/tensor.agenthub.uk\/?p=416"},"modified":"2024-04-29T07:47:33","modified_gmt":"2024-04-29T07:47:33","slug":"sklearn%e5%a6%82%e4%bd%95%e6%b1%82%e8%a7%a3%e7%ba%bf%e6%80%a7%e5%9b%9e%e5%bd%92","status":"publish","type":"post","link":"https:\/\/tensorzen.blog\/?p=416","title":{"rendered":"sklearn\u5982\u4f55\u6c42\u89e3\u7ebf\u6027\u56de\u5f52"},"content":{"rendered":"\n

sklearn\u662f\u5982\u4f55\u6c42\u89e3\u7ebf\u6027\u56de\u5f52\u95ee\u9898\u7684\uff1f\u6709\u6ca1\u6709\u8131\u53e3\u800c\u51fa\u68af\u5ea6\u4e0b\u964d\uff0c\u8d77\u7801\u6211\u81ea\u5df1\u4e0b\u610f\u8bc6\u7684\u662f\u8fd9\u4e48\u8ba4\u4e3a\u7684\uff0c\u76f4\u5230\u6709\u4e00\u6b21\u65e0\u610f\u4e2d\u770b\u5230sklearn\u7684\u7528\u6237\u624b\u518c\uff0c1.1.1.2 \u4e00\u884c\u5c0f\u5c0f\u7684\u5907\u6ce8\uff1a<\/p>\n\n\n\n

The least squares solution is computed using the singular value decomposition of X.<\/p>\n\n\n\n

\u539f\u6765\u5b83\u662f\u7528SVD\u6765\u6c42\u89e3\u7684\uff0c\u4e8e\u662f\u6211\u4eec\u6765\u804a\u4e0b\u4e3a\u4ec0\u4e48SVD\u53ef\u4ee5\u5e72\u8fd9\u4e2a\u6d3b\u3002<\/p>\n\n\n\n

\u7ebf\u6027\u56de\u5f52\u95ee\u9898\u5c31\u662f\u6c42\u89e3\u6700\u5c0f\u4e8c\u4e58(least squares)\uff0c\u7ed9\u5b9a\u4e00\u7ec4\u5148\u884c\u65b9\u7a0b\u7ec4$Ax = b$\uff0c\u6c42$x$\u662f\u4ec0\u4e48\uff0c\u8fd9\u4e2a\u65b9\u7a0b\u662f\u8d85\u5b9a\u65b9\u7a0b\uff08overdetermined\uff09\uff0c\u5c31\u662f\u8bf4\u65b9\u7a0b\u7ec4\u7684\u6570\u91cf\u5927\u4e8e\u672a\u77e5\u53d8\u91cf\u6570\u3002\u6700\u5c0f\u4e8c\u4e58\u901a\u8fc7\u6700\u5c0f\u5316$||b – Ax||^2$\u5f97\u5230\u7684$x$\u4f5c\u4e3a\u89e3\uff0c\u5176\u4e2d$A$\u662f\u81ea\u53d8\u91cf\uff0c$b$\u662f\u56e0\u53d8\u91cf\u3002\u5728\u673a\u5668\u5b66\u6821\u9886\u57df\u901a\u5e38\u7528$x$\u6765\u6807\u8bb0\u81ea\u53d8\u91cf\uff08features\uff09\uff0c$y$\u6807\u8bb0\u56e0\u53d8\u91cf\uff08labels\uff09\uff0c\u65b9\u4fbf\u8ba8\u8bba\u8fd8\u662f\u7528$x$\u548c$A$\uff0c\u6700\u5c0f\u4e8c\u4e58\u7684\u76ee\u6807<\/p>\n\n\n\n

$$\\min \\sum_{i=1}^{m} [b_i – (Ax)_i]^2$$<\/p>\n\n\n\n

\u5176\u4e2d$A$\u662f\u4e00\u4e2a$m \\times n$\u7684\u77e9\u9635\uff0c\u8868\u793a\u6211\u4eec\u6709$m$\u4e2a\u6837\u672c$n$\u4e2a\u672a\u77e5\u6570<\/p>\n\n\n\n

\u4e0b\u9762\u7684\u8ba8\u8bba\uff0c\u6211\u4eec\u5047\u8bbe$A$\u662f\u6ee1\u79e9\u7684\u3002<\/p>\n\n\n\n

\u4efb\u610f\u77e9\u9635\u90fd\u53ef\u4ee5\u8fdb\u884cSVD\u5206\u89e3\uff0c\u4e0d\u8981\u95ee\u4e3a\u4ec0\u4e48\uff0c\u5c31\u662f\u8fd9\u4e48\u725bx\uff0c\u7ecf\u8fc7SVD\u5206\u89e3\u53ef\u4ee5\u628a$m\\times n$\u7684\u77e9\u9635$A$\u5206\u89e3\u6210\u4e09\u4e2a\u77e9\u9635<\/p>\n\n\n\n

$$A = U\\Sigma V^T$$<\/p>\n\n\n\n

\u5176\u4e2d$U \\in R^{m \\times m}$\u662f\u4e00\u4e2a\u6b63\u4ea4\u77e9\u9635,$\\Sigma \\in R^{m \\times n}$\u7684\u4e3b\u5bf9\u89d2\u7ebf\u5143\u7d20\u4ece\u5de6\u4e0a\u89d2\u5230\u53f3\u4e0b\u89d2\u4f9d\u6b21\u53d8\u5c0f\uff0c$A$\u662f\u6ee1\u79e9\u7684\uff0c\u6240\u4ee5$\\Sigma$\u4e5f\u662f\u6ee1\u79e9\u7684\uff0c$\\Sigma$\u91cc\u7684\u5143\u7d20$\\sigma$\u662f$A$\u7684\u5947\u5f02\u503c\uff0c\u5b83\u7684\u5e73\u65b9\u662f$A^TA$\u7684\u7279\u5f81\u503c\u6216\u8005$AA^T$\u7684top n\u5927\u7684\u7279\u5f81\u503c\uff0c$V \\in R^{n\\times n}$\u7684\u6b63\u4ea4\u77e9\u9635\u3002<\/p>\n\n\n\n

$U$\u548c$V$\u662f\u4ec0\u4e48\u5462\uff1f\u6309Prof. Gilbert Strang\u7684\u8bf4\u6cd5”I like A transpose A”\uff0c\u4e8e\u662f<\/p>\n\n\n\n

$$A^TA = V \\Sigma^T U^T U\\Sigma V^T = V \\Sigma^2 V^T$$<\/p>\n\n\n\n

$U \\in R^{m \\times m}$\u662f\u6b63\u4ea4\u77e9\u9635\uff0c\u5b83\u6ee1\u8db3$U^TU =UU^T = \\textbf{I}$,$U^T = U^{-1}$,\u7b49\u5f0f\u53f3\u8fb9\u91cd\u5199\u4e00\u4e0b\u5f97\u5230$V \\Sigma ^2 V^{-1}$,\u8fd9\u5176\u5b9e\u662f\u7279\u5f81\u5206\u89e3\u7684\u5b9a\u4e49\uff0c\u770b\u4e0b\u7ef4\u57fa\u767e\u79d1\u7684\u5b9a\u4e49\uff1a<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

\u6240\u4ee5$V$\u7684\u6bcf\u4e00\u5217\u662f$AA^T$\u7684\u7279\u5f81\u5411\u91cf\uff0c$\\Sigma^2$\u91cc\u88c5\u7684\u662f\u5bf9\u5e94\u7684\u7279\u5f81\u503c\u3002<\/p>\n\n\n\n

\u518d\u6765\u7b97\u4e00\u4e0b$AA^T$<\/p>\n\n\n\n

$$AA^T = U\\Sigma V^T V \\Sigma ^T U^T = U \\Sigma ^2 U^T$$<\/p>\n\n\n\n

\u6240\u4ee5$U$\u662f$AA^T$\u7684\u7279\u5f81\u5411\u91cf\u7ec4\u6210\u7684\u77e9\u9635\u3002<\/p>\n\n\n\n

\u6765\u770b\u4e0bSVD\u7684\u7528\u6765\u6c42\u89e3least square\u7684\u8fc7\u7a0b<\/p>\n\n\n\n

$$Ax = b$$<\/p>\n\n\n\n

\u4e24\u8fb9\u540c\u65f6\u4e58$A^T$<\/p>\n\n\n\n

$$A^TAx = A^Tb$$<\/p>\n\n\n\n

\u5bf9$A$\u8fdb\u884cSVD\u5206\u89e3<\/p>\n\n\n\n

$$V\\Sigma ^T U^T U \\Sigma V^T x = V\\Sigma ^TU^T b$$<\/p>\n\n\n\n

\u628a\u90a3\u4e9b\u6b63\u4ea4\u77e9\u9635\u6574\u5408\u4e0b<\/p>\n\n\n\n

$$V \\Sigma ^2 V^T x = V \\Sigma^ T U^T b$$<\/p>\n\n\n\n

\u5de6\u4e58$V\\Sigma^{-1}$<\/p>\n\n\n\n

$$\\Sigma V^Tx = U^Tb$$<\/p>\n\n\n\n

\u56e0\u4e3a$\\Sigma$\u662f\u5bf9\u89d2\u77e9\u9635\uff0c$\\Sigma^T = \\Sigma$, $\\Sigma^{-1} = \\frac{1}{\\Sigma}$\uff0c\u518d\u5de6\u4e58$V\\Sigma^{-1}$<\/p>\n\n\n\n

$$V \\Sigma^{-1}\\Sigma V^T x = V \\Sigma^{-1}U^Tb$$<\/p>\n\n\n\n

\u4e8e\u662f\u6700\u540e\u5269\u4e0b<\/p>\n\n\n\n

$$x = V\\Sigma ^{-1} U^T b$$<\/p>\n\n\n\n

\u5176\u4e2d$V$\u548c$U$\u53ef\u4ee5\u7b97\u51fa\u6765\uff0c$\\Sigma^{-1}$\u5c31\u662f$\\frac{1}{\\Sigma}$\u3002<\/p>\n\n\n\n

\u5982\u679c$A$\u662f\u975e\u5947\u5f02\u77e9\u9635(no-singular matrix)\uff0c\u5373\u65b9\u7a0b\u662fwell-determined\uff0c\u77e9\u9635\u53ef\u9006\uff0c\u6253\u773c\u4e00\u770b\u5c31\u77e5\u9053\u65b9\u7a0b\u7684\u89e3<\/p>\n\n\n\n

$$x = A^{-1}b$$<\/p>\n\n\n\n

\u4f46\u662f\u5f53$A$\u662f\u4e0d\u53ef\u9006\u7684\u65f6\u5019\u6839\u636e\u4e0a\u9762\u7684\u63a8\u5bfc\u8fc7\u7a0b$V\\Sigma ^{-1} U^T$\u627f\u62c5\u4e86$A^{-1}$\u7684\u4efb\u52a1\uff0c\u4e8e\u662f\u6211\u4eec\u53eb\u5b83\u5047\u7684\u9006\u77e9\u9635\uff0c\u4f2a\u9006\u77e9\u9635(pseudoinverse)\uff0c\u6807\u8bb0\u4e3a$A^{+}$\uff0c\u7ef4\u57fa\u767e\u79d1<\/a>\u8bf7\u81ea\u884c\u9605\u8bfb\u3002<\/p>\n\n\n\n

\u5927\u6982\u5c31\u662f\u8fd9\u4e48\u4e2a\u8fc7\u7a0b\uff0c\u6b64\u81f4\u656c\u793c\u3002<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

sklearn\u662f\u5982\u4f55\u6c42\u89e3\u7ebf\u6027\u56de\u5f52\u95ee\u9898\u7684\uff1f\u6211\u4e00\u76f4\u4ee5\u4e3a\u662f\u91c7\u7528\u68af\u5ea6\u4e0b\u964d\uff0c\u663e\u7136\u4e0d\u662f\u7684\u3002<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,4],"tags":[],"class_list":["post-416","post","type-post","status-publish","format-standard","hentry","category-base","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts\/416","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=416"}],"version-history":[{"count":18,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts\/416\/revisions"}],"predecessor-version":[{"id":438,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=\/wp\/v2\/posts\/416\/revisions\/438"}],"wp:attachment":[{"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=416"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=416"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tensorzen.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=416"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}