爬取豆瓣电影top250，出现以下报错：
2018-08-11 22:02:16 [scrapy.core.engine] INFO: Spider opened 2018-08-11 22:02:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2018-08-11 22:02:16 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/robots.txt> (referer: None) 2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/top250> (referer: None) 2018-08-11 22:02:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not allowed 2018-08-11 22:02:17 [scrapy.core.engine] INFO: Closing spider (finished)
防止反爬机制，伪装user_agent
【1】打开豆瓣top250 ： https://movie.douban.com/top250
【2】F12 打开控制台->刷新页面 ->Network->请求头部找到 User-Agent
在scrapy项目中找到settings.py的 USER_AGENT = ' ' （把注释去掉，加以下内容）
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3493.3 Safari/537.36'
重新执行即可
scrapy crawl douban_spider
转载于:https://www.cnblogs.com/QW-lzm/p/9461375.html
