{"id":2525,"date":"2018-02-24T15:16:05","date_gmt":"2018-02-24T07:16:05","guid":{"rendered":"http:\/\/www.xttblog.com\/?p=2525"},"modified":"2018-02-24T15:16:05","modified_gmt":"2018-02-24T07:16:05","slug":"headless-chrome-crawler-%e6%95%99%e7%a8%8b","status":"publish","type":"post","link":"https:\/\/www.xttblog.com\/?p=2525","title":{"rendered":"headless-chrome-crawler \u6559\u7a0b"},"content":{"rendered":"<p>\n\t<a href=\"https:\/\/github.com\/yujiosaka\/headless-chrome-crawler\" name=\"\u722c\u866b\" target=\"_blank\" title=\"\u722c\u866b\">headless-chrome-crawler<\/a> \u662f\u4e00\u4e2a\u81ea\u5e26 JavaScript \u6267\u884c\u73af\u5883\u7684\u722c\u866b\u63d2\u4ef6\u3002\u5b83\u652f\u6301\u5206\u5e03\u5f0f\uff0c\u662f\u4e00\u6b3e\u5206\u5e03\u5f0f\u722c\u866b\u3002\u5b83\u80fd\u89e3\u51b3 AngularJS\u3001vue.js \u7b49\u8fd9\u4e9b\u73b0\u4ee3\u5316\u7684\u524d\u7aef\u6846\u67b6\u7f16\u5199\u7684\u7f51\u7ad9\u7684\u722c\u866b\u95ee\u9898\u3002\u672c\u6587\u5c06\u8be6\u7ec6\u7684\u4ecb\u7ecd\u5b83\u7684\u76f8\u5173\u6559\u7a0b\u3002\n<\/p>\n<p>\n\t\u524d\u9762\u6211\u5199\u8fc7\u300a<a href=\"https:\/\/www.xttblog.com\/?p=1309\" title=\"Webmagic(\u722c\u866b)\u6293\u53d6\u65b0\u6d6a\u535a\u5ba2\u6848\u4f8b - \u4e1a\u4f59\u8349\">Webmagic(\u722c\u866b)\u6293\u53d6\u65b0\u6d6a\u535a\u5ba2\u6848\u4f8b<\/a>\u300b<a href=\"https:\/\/www.xttblog.com\/?p=1309\" title=\"Webmagic(\u722c\u866b)\u6293\u53d6\u65b0\u6d6a\u535a\u5ba2\u6848\u4f8b - \u4e1a\u4f59\u8349\">Webmagic<\/a>&nbsp;\u662f\u4e00\u4e2a\u57fa\u4e8e java \u5b9e\u73b0\u7684\u722c\u866b\u6846\u67b6\uff0c\u5173\u4e8e\u5b83\u7684\u76f8\u5173\u6559\u7a0b\u90fd\u53ef\u4ee5\u5728\u6211\u7684\u535a\u5ba2\u4e2d\u627e\u5230\u3002\n<\/p>\n<h2>\n\t\u7279\u5f81<br \/>\n<\/h2>\n<ul>\n<li>\n\t\t<font><font>\u5206\u5e03\u5f0f\u6293\u53d6<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u914d\u7f6e\u5e76\u53d1\u6027\uff0c\u5ef6\u8fdf\u548c\u91cd\u8bd5<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u652f\u6301<\/font><\/font><a href=\"https:\/\/en.wikipedia.org\/wiki\/Depth-first_search\" rel=\"nofollow\"><font><font>\u6df1\u5ea6\u4f18\u5148\u641c\u7d22<\/font><\/font><\/a><font><font>\u548c<\/font><\/font><a href=\"https:\/\/en.wikipedia.org\/wiki\/Breadth-first_search\" rel=\"nofollow\"><font><font>\u5e7f\u5ea6\u4f18\u5148\u641c\u7d22<\/font><\/font><\/a><font><font>\u7b97\u6cd5<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u53ef\u63d2\u62d4\u7684\u7f13\u5b58\u5b58\u50a8\uff0c\u5982<\/font><\/font><a href=\"https:\/\/redis.io\/\" rel=\"nofollow\"><font><font>Redis<\/font><\/font><\/a>\n\t<\/li>\n<li>\n\t\t<font><font>\u652f\u6301<\/font><font>\u5bfc\u51fa\u7ed3\u679c\u7684<\/font><\/font><a href=\"https:\/\/tools.ietf.org\/html\/rfc4180\" rel=\"nofollow\"><font><font>CSV<\/font><\/font><\/a><font><font>\u548c<\/font><\/font><a href=\"http:\/\/jsonlines.org\/\" rel=\"nofollow\"><font><font>JSON\u884c<\/font><\/font><\/a>\n\t<\/li>\n<li>\n\t\t<font><font>\u5728\u6700\u5927\u8bf7\u6c42\u65f6\u6682\u505c\u5e76\u5728\u4efb\u4f55\u65f6\u5019\u6062\u590d<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u81ea\u52a8<\/font><font>\u63d2\u5165<\/font><\/font><a href=\"https:\/\/jquery.com\/\" rel=\"nofollow\"><font><font>jQuery<\/font><\/font><\/a><font><font>\u8fdb\u884c\u522e\u53d6<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u4fdd\u5b58\u6293\u53d6\u8bc1\u636e\u7684\u622a\u56fe<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u4eff\u771f\u8bbe\u5907\u548c\u7528\u6237\u4ee3\u7406<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u4f18\u5148\u961f\u5217\u63d0\u9ad8\u6293\u53d6\u6548\u7387<\/font><\/font>\n\t<\/li>\n<li>\n\t\t<font><font>\u670d\u4ece<\/font><\/font><a href=\"https:\/\/developers.google.com\/search\/reference\/robots_txt\" rel=\"nofollow\"><font><font>robots.txt<\/font><\/font><\/a>\n\t<\/li>\n<li>\n\t\t<font><font>\u8ddf\u968f<\/font><\/font><a href=\"https:\/\/www.sitemaps.org\/\" rel=\"nofollow\"><font><font>sitemap.xml<\/font><\/font><\/a>\n\t<\/li>\n<\/ul>\n<h2>\n\t<font><font>\u5b89\u88c5<\/font><\/font><br \/>\n<\/h2>\n<p>\n\theadless-chrome-crawler \u6709\u4e24\u79cd\u5b89\u88c5\u65b9\u5f0f\uff0c\u5982\u4e0b\uff1a\n<\/p>\n<pre class=\"brush:bash;\">\r\nyarn add headless-chrome-crawler\r\n# or &quot;npm i headless-chrome-crawler&quot;<\/pre>\n<p>\n\t\u722c\u884c\u5668\u5305\u542b<a href=\"https:\/\/github.com\/GoogleChrome\/puppeteer\">Puppeteer<\/a>\u3002\u5728\u5b89\u88c5\u8fc7\u7a0b\u4e2d\uff0c\u5b83\u4f1a\u81ea\u52a8\u4e0b\u8f7d\u6700\u65b0\u7248\u672c\u7684Chromium\u3002\n<\/p>\n<h2>\n\t\u7528\u6cd5<br \/>\n<\/h2>\n<p>\n\t\u5b83\u7684\u7528\u6cd5\u5f88\u7b80\u5355\uff0c\u5982\u4e0b\uff1a\n<\/p>\n<pre class=\"brush:jscript;\">\r\nconst HCCrawler = require(&#039;headless-chrome-crawler&#039;);\r\n\r\nHCCrawler.launch({\r\n  \/\/ Function to be evaluated in browsers\r\n  evaluatePage: (() =&gt; ({\r\n    title: $(&#039;title&#039;).text(),\r\n  })),\r\n  \/\/ Function to be called with evaluated results from browsers\r\n  onSuccess: (result =&gt; {\r\n    console.log(result);\r\n  }),\r\n})\r\n  .then(crawler =&gt; {\r\n    \/\/ Queue a request\r\n    crawler.queue(&#039;https:\/\/example.com\/&#039;);\r\n    \/\/ Queue multiple requests\r\n    crawler.queue(['https:\/\/example.net\/', 'https:\/\/example.org\/']);\r\n    \/\/ Queue a request with custom options\r\n    crawler.queue({\r\n      url: &#039;https:\/\/example.com\/&#039;,\r\n      \/\/ Emulate a tablet device\r\n      device: &#039;Nexus 7&#039;,\r\n      \/\/ Enable screenshot by passing options\r\n      screenshot: {\r\n        path: &#039;.\/tmp\/example-com.png&#039;\r\n      },\r\n    });\r\n    crawler.onIdle() \/\/ Resolved when no queue is left\r\n      .then(() =&gt; crawler.close()); \/\/ Close the crawler\r\n  });<\/pre>\n<p>\n\t\u53e6\u5916 github \u4e0a\u8fd8\u63d0\u4f9b\u4e86\u5f88\u591a\u6848\u4f8b\u548c api \u6587\u6863\u7684\u4f7f\u7528\u3002\u66f4\u591a\u7528\u6cd5\u8bf7\u79fb\u6b65\u5230 github \u4e0a\u81ea\u884c\u5b66\u4e60\u3002\n<\/p>\n<section>\n<article>\n\t\t<!--StartFragment--><!--EndFragment--><br \/>\n\t<\/article>\n<\/section>\n<!-- <audio autoplay=\"autoplay\" height=\"100\" width=\"100\" hidden=\"true\"><source src=\"https:\/\/c.coka.la\/orAvPR.mp3\" type=\"audio\/mp3\"\/><\/audio><img decoding=\"async\" src=\"https:\/\/images.weserv.nl\/?url=https:\/\/ws3.sinaimg.cn\/large\/8fa5dcfcgy1g4em3rzar5j20ib089ae0.jpg\" alt=\"\u4e1a\u4f59\u8349\u516c\u4f17\u53f7\"\/> --><p><!-- <img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/xmt1139057136\/xttblog\/master\/8fa5dcfcgy1g4em3rzar5j20ib089ae0.jpg\" alt=\"\u4e1a\u4f59\u8349\u516c\u4f17\u53f7\"\/>--><img decoding=\"async\" src=\"https:\/\/saas.51autoparts.cn\/api\/admin\/sys-file\/oss\/file?fileName=84db51f8561e448f9a4fda2eff0811f9.jpg\" alt=\"\u4e1a\u4f59\u8349\u516c\u4f17\u53f7\"\/><\/p><p>\u6700\u540e\uff0c\u6b22\u8fce\u5173\u6ce8\u6211\u7684\u4e2a\u4eba\u5fae\u4fe1\u516c\u4f17\u53f7\uff1a\u4e1a\u4f59\u8349\uff08yyucao\uff09\uff01<b>\u53ef\u52a0\u4f5c\u8005\u5fae\u4fe1\u53f7\uff1a<font color=\"red\">xttblog2<\/font>\u3002\u5907\u6ce8\uff1a\u201c1\u201d\uff0c\u6dfb\u52a0\u535a\u4e3b\u5fae\u4fe1\u62c9\u4f60\u8fdb\u5fae\u4fe1\u7fa4\u3002\u5907\u6ce8\u9519\u8bef\u4e0d\u4f1a\u540c\u610f\u597d\u53cb\u7533\u8bf7\u3002<\/b>\u518d\u6b21\u611f\u8c22\u60a8\u7684\u5173\u6ce8\uff01\u540e\u7eed\u6709\u7cbe\u5f69\u5185\u5bb9\u4f1a\u7b2c\u4e00\u65f6\u95f4\u53d1\u7ed9\u60a8\uff01\u539f\u521b\u6587\u7ae0\u6295\u7a3f\u8bf7\u53d1\u9001\u81f3532009913@qq.com\u90ae\u7bb1\u3002\u5546\u52a1\u5408\u4f5c\u4e5f\u53ef\u6dfb\u52a0\u4f5c\u8005\u5fae\u4fe1\u8fdb\u884c\u8054\u7cfb\uff01<\/p><p>\u672c\u6587\u539f\u6587\u51fa\u5904\uff1a<a href=\"https:\/\/www.xttblog.com\">\u4e1a\u4f59\u8349<\/a>\uff1a &raquo; <a href=\"https:\/\/www.xttblog.com\/?p=2525\">headless-chrome-crawler \u6559\u7a0b<\/a><\/p><!-- <div class=\"bd-reward-stl\"><button id=\"bdRewardBtn\"><span><\/span><\/button><\/div> -->","protected":false},"excerpt":{"rendered":"<p>headless-chrome-crawler \u662f\u4e00\u4e2a\u81ea\u5e26 JavaScript \u6267\u884c\u73af\u5883\u7684\u722c\u866b\u63d2\u4ef6\u3002\u5b83\u652f\u6301\u5206 [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,5],"tags":[110,1555,1048,108,1031],"class_list":["post-2525","post","type-post","status-publish","format-standard","hentry","category-html5","category-java","tag-jsoup","tag-nodejs","tag-webmagic","tag-108","tag-1031"],"_links":{"self":[{"href":"https:\/\/www.xttblog.com\/index.php?rest_route=\/wp\/v2\/posts\/2525","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.xttblog.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.xttblog.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.xttblog.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.xttblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2525"}],"version-history":[{"count":1,"href":"https:\/\/www.xttblog.com\/index.php?rest_route=\/wp\/v2\/posts\/2525\/revisions"}],"predecessor-version":[{"id":2526,"href":"https:\/\/www.xttblog.com\/index.php?rest_route=\/wp\/v2\/posts\/2525\/revisions\/2526"}],"wp:attachment":[{"href":"https:\/\/www.xttblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2525"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.xttblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2525"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.xttblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2525"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}