引言

在Scrapy中，在很多種情況下，需要一層層地進行爬取網頁數據，就是基于url爬取網頁，然后在從網頁中提取url，繼續(xù)爬取，循環(huán)往復。

本文將講述一個在迭代爬取中，只能爬取第一層網頁的問題。

問題的提出

scrapy crawl enrolldata
Scrapy代碼執(zhí)行結果輸出如下：
“`
2018-05-06 17:23:06 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: enrolldata)
2018-05-06 17:23:06 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 3.6.1 (default, Apr 24 2017, 23:31:02) - [GCC 6.2.0 20161005], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Linux-4.15.0-20-generic-x86_64-with-debian-buster-sid
2018-05-06 17:23:06 [scrapy.crawler] INFO: Overridden settings: {‘BOT_NAME’: ‘enrolldata’, ‘CONCURRENT_REQUESTS’: 60, ‘CONCURRENT_REQUESTS_PER_IP’: 60, ‘DEPTH_LIMIT’: 5, ‘NEWSPIDER_MODULE’: ‘enrolldata.spiders’, ‘SPIDER_MODULES’: [‘enrolldata.spiders’]}
2018-05-06 17:23:06 [scrapy.middleware] INFO: Enabled extensions:
[‘scrapy.extensions.corestats.CoreStats’,
‘scrapy.extensions.telnet.TelnetConsole’,
‘scrapy.extensions.memusage.MemoryUsage’,
‘scrapy.extensions.logstats.LogStats’]
2018-05-06 17:23:06 [scrapy.middleware] INFO: Enabled downloader middlewares:
[‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware’,
‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware’,
‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware’,
‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware’,
‘scrapy.downloadermiddlewares.retry.RetryMiddleware’,
‘scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware’,
‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware’,
‘scrapy.downloadermiddlewares.redirect.RedirectMiddleware’,
‘scrapy.downloadermiddlewares.cookies.CookiesMiddleware’,
‘scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware’,
‘scrapy.downloadermiddlewares.stats.DownloaderStats’]
2018-05-06 17:23:06 [scrapy.middleware] INFO: Enabled spider middlewares:
[‘scrapy.spidermiddlewares.httperror.HttpErrorMiddleware’,
‘scrapy.spidermiddlewares.offsite.OffsiteMiddleware’,
‘scrapy.spidermiddlewares.referer.RefererMiddleware’,
‘scrapy.spidermiddlewares.urllength.UrlLengthMiddleware’,
‘scrapy.spidermiddlewares.depth.DepthMiddleware’]
2018-05-06 17:23:06 [scrapy.middleware] INFO: Enabled item pipelines:
[‘enrolldata.pipelines.EnrolldataPipeline’]
2018-05-06 17:23:06 [scrapy.core.engine] INFO: Spider opened
open spider ………..pipeline
2018-05-06 17:23:06 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-05-06 17:23:06 [py.warnings] WARNING: /home/bladestone/codebase/python36env/lib/python3.6/site-packages/scrapy/spidermiddlewares/offsite.py:59: URLWarning: allowed_domains accepts only domains, not URLs. Ignoring URL entry http://www.heao.gov.cn/ in allowed_domains.
warnings.warn(“allowed_domains accepts only domains, not URLs. Ignoring URL entry %s in allowed_domains.” % domain, URLWarning)

2018-05-06 17:23:06 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
end of start requests
2018-05-06 17:23:14 [scrapy.core.engine] DEBUG: Crawled (200)

 -*- coding: utf-8 -*-
import scrapy
from enrolldata.items import EnrolldataItem

from scrapy.http import FormRequest

class SchoolspiderSpider(scrapy.Spider):
    name = 'enrolldata'
    cookies = {}
    allowed_domains = ['http://www.heao.gov.cn/']
    start_urls = ['http://www.heao.gov.cn/JHCX/PZ/enrollplan/SchoolList.aspx']
   .............
   def start_requests(self):
        formdata = {}
        formdata['PagesUpDown$edtPage'] = '1'
        formdata['__EVENTTARGET'] = 'PagesUpDown$lbtnGO'
        formdata['__EVENTARGUMENT'] = ''
        formdata['__VIEWSTATE'] = '/wEPDwUKMjA1MTU4MDA1Ng9kFgICBQ9kFgICAQ8PFggeDGZDdXJyZW50UGFnZQIBHhFmVG90YWxSZWNvcmRDb3VudAK4ER4KZlBhZ2VDb3VudAKVAR4JZlBhZ2VTaXplAg9kZGSI36vb/TsBmDT8pwwx37ajH1x0og=='
        formdata['__VIEWSTATEGENERATOR'] = 'AABB4DD8'
        formdata['__EVENTVALIDATION'] = '/wEWBQLYvvTTCwK2r/yJBQK6r/CJBgLqhPDLCwLQ0r3uCMy0KhJCAT8jebTQL0eNdj7uk4L5'
        for i in range(1, 2):
            formdata['PagesUpDown$edtPage'] = str(i)
            yield FormRequest(url=self.start_urls[0], headers=self.headers, formdata=formdata, callback=self.parse_school)
        print("end of start requests")

    def parse(self, response):
        print("parse method is invoked")
        pass

    def parse_school(self, response):
        print("parse school data.....")
        urls = response.xpath('//*[@id="SpanSchoolList"]/div/div[2]/ul/li/a/@href').extract();
        print("print out all the matched urls")
        print(urls)

        for url in urls:
            request_url = self.base_url + url
            print("request_url in major:" + request_url)
            yield scrapy.Request(request_url, headers=self.request_headers,  cookies=self.cookies, callback=self.parse_major_enroll, meta=self.meta)
......

代碼沒有報錯，只是輸出了第一層的Web的爬取結果。但是第二層沒有執(zhí)行爬取。

問題分析

從日志來進行分析，沒有發(fā)現錯誤信息；第一層代碼爬取正確，但是第二層web爬取，沒有被執(zhí)行，代碼的編寫應該沒有問題的。

那問題是什么呢？會不會代碼沒有被執(zhí)行呢？通過添加日志，但是對應的代碼并沒有執(zhí)行，日志也被正常輸出。是不是被過濾或者攔截了，從而代碼沒有被執(zhí)行？

經過代碼審查之后，發(fā)現allowed_domains設置的問題，由于起設置不正確，導致其余的鏈接被直接過濾了。

關于allowed_domains需要是一組域名，而非一組urls。

問題的解決

需要將之前的domain name修改一下：

allowed_domains = [‘http://www.heao.gov.cn/‘]

將起修改為：

allowed_domains = [‘heao.gov.cn']

重新執(zhí)行爬蟲，發(fā)現多個層次是可以被正確爬取的。

總結

關于scrapy是一整套的解決方案，其中很多的設置和配置需要通過不同的實例來反復理解和應用的，才能如魚得水，庖丁解牛般快速定位問題。

以上為個人經驗，希望能給大家一個參考，也希望大家多多支持腳本之家。

最新評論