在scrapy中使用phantomJS實現(xiàn)異步爬取的方法

更新時間：2018年12月17日 11:16:02 作者：郎總

今天小編就為大家分享一篇在scrapy中使用phantomJS實現(xiàn)異步爬取的方法，具有很好的參考價值，希望對大家有所幫助。一起跟隨小編過來看看吧

使用selenium能夠非常方便的獲取網(wǎng)頁的ajax內容，并且能夠模擬用戶點擊和輸入文本等諸多操作，這在使用scrapy爬取網(wǎng)頁的過程中非常有用。

網(wǎng)上將selenium集成到scrapy的文章很多，但是很少有能夠實現(xiàn)異步爬取的，下面這段代碼就重寫了scrapy的downloader，同時實現(xiàn)了selenium的集成以及異步。

使用時需要PhantomJSDownloadHandler添加到配置文件的DOWNLOADER中。

# encoding: utf-8
from __future__ import unicode_literals
 
from scrapy import signals
from scrapy.signalmanager import SignalManager
from scrapy.responsetypes import responsetypes
from scrapy.xlib.pydispatch import dispatcher
from selenium import webdriver
from six.moves import queue
from twisted.internet import defer, threads
from twisted.python.failure import Failure
 
 
class PhantomJSDownloadHandler(object):
 
 def __init__(self, settings):
  self.options = settings.get('PHANTOMJS_OPTIONS', {})
 
  max_run = settings.get('PHANTOMJS_MAXRUN', 10)
  self.sem = defer.DeferredSemaphore(max_run)
  self.queue = queue.LifoQueue(max_run)
 
  SignalManager(dispatcher.Any).connect(self._close, signal=signals.spider_closed)
 
 def download_request(self, request, spider):
  """use semaphore to guard a phantomjs pool"""
  return self.sem.run(self._wait_request, request, spider)
 
 def _wait_request(self, request, spider):
  try:
   driver = self.queue.get_nowait()
  except queue.Empty:
   driver = webdriver.PhantomJS(**self.options)
 
  driver.get(request.url)
  # ghostdriver won't response when switch window until page is loaded
  dfd = threads.deferToThread(lambda: driver.switch_to.window(driver.current_window_handle))
  dfd.addCallback(self._response, driver, spider)
  return dfd
 
 def _response(self, _, driver, spider):
  body = driver.execute_script("return document.documentElement.innerHTML")
  if body.startswith("<head></head>"): # cannot access response header in Selenium
   body = driver.execute_script("return document.documentElement.textContent")
  url = driver.current_url
  respcls = responsetypes.from_args(url=url, body=body[:100].encode('utf8'))
  resp = respcls(url=url, body=body, encoding="utf-8")
 
  response_failed = getattr(spider, "response_failed", None)
  if response_failed and callable(response_failed) and response_failed(resp, driver):
   driver.close()
   return defer.fail(Failure())
  else:
   self.queue.put(driver)
   return defer.succeed(resp)
 
 def _close(self):
  while not self.queue.empty():
   driver = self.queue.get_nowait()
   driver.close()

以上這篇在scrapy中使用phantomJS實現(xiàn)異步爬取的方法就是小編分享給大家的全部內容了，希望能給大家一個參考，也希望大家多多支持腳本之家。

您可能感興趣的文章: