scrapy Splash Request设置请求头
1. 在lua里直接设置：splash:set_user_agent(“Mozilla/5.0”)
2. 写在middleware里面：
class HeaderMiddleware(object):
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_request(self, request, spider):
request.headers["user-agent"] = UserAgent().random
request.headers["authority"] = 'www.ame.com'
request.headers["accept-language"] = 'en;q=0.8'
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
再在爬虫的customer_settings里添加：
