scrapy 带认证的代理

佚名 6年前 (2020-01-17) 随笔 2258人围观抢沙发百度已收录

官方方法：

from w3lib.http import basic_auth_header

class CustomProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = "https://<PROXY_IP_OR_URL>:<PROXY_PORT>"
        request.headers['Proxy-Authorization'] = basic_auth_header(
            '<PROXY_USERNAME>', '<PROXY_PASSWORD>')

SRE实战互联网时代守护先锋，助力企业售后服务体系运筹帷幄！一键直达领取阿里云限量特价优惠。

DOWNLOADER_MIDDLEWARES = {
    '<PROJECT_NAME>.middlewares.CustomProxyMiddleware': 350,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

来源： https://support.scrapinghub.com/support/solutions/articles/22000219743-using-a-custom-proxy-in-a-scrapy-spider

经过测试，不带认证的这么写也没有问题