用Scrapy Splash 来抓取渲染后的html页面

很多复杂的网页都是用javascript来对网页进行填充，这样用request的body和在浏览器中看到的不一样啊。这个时候splash就可以使用了，它是提供一个轻量级的api，传给它网址，它返回网页内容。这样就OK了

1.安装splash

要先安装docker

docker pull registry.docker-cn.com/scrapinghub/splash   #从docker镜像中拉取splash实例
docker run -p 8050:8050 scrapinghub/splash  #启动splash实例

现在docker 可以用国内的源了，所以要加上registry.docker-cn.com来加快速度

还需要安装个scrapy-splash

sudo pip3 install scrapy-splash

2. setting 设置

SPLASH_URL = 'http://localhost:8050'  

DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

3. spider中使用

import scrapy
import logging
import re
import datetime
from tutorial import settings
from scrapy_splash import SplashRequest

class JanDanSpider(scrapy.Spider):
    name = "jandan"

    def start_requests(self):
        #start_url = 'http://jandan.net/ooxx'
        start_url = 'http://www.baidu.com'
        headers = {
            'Connection' : 'keep - alive',
            'User-Agent' : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36'
        }

        yield SplashRequest(url=start_url, callback = self.parse, args={'wait': 1.0})


    def parse(self, response):

        # --------------------------------body image url--------------------------------------------
        image_urls = response.css('img::attr("src")').extract()


        new_image_urls = []
        for i in range(len(image_urls)):
            new_image_urls.append('http://' + image_urls[i][2:])

        pass

主要这里用了SplashRequest，加了一个等待时间，callback回来就跟默认的一样了。进行爬虫的其他处理。。。

http://www.waitingfy.com/archives/3302

scrapy | RSS 2.0 | Respond | Trackback |