Proxy Integration with Scrapy Proxy Middleware

A residential proxy serves as a mediator, allowing users to get an IP address from an authentic desktop or mobile device connected to a local network. Due to its origin, residential proxies are a perfect match for overcoming geo-restrictions, bypassing CAPTCHAs, managing multiple accounts, and conducting web testing with the CapSolver platform.

Decodo offers top-notch residential proxies with an extensive IP pool of over 55M IPs across 195+ locations. With an unparalleled responsive rate, clocking in at under 0.6 seconds, a success rate of 99.86%, and an affordable entry point with Pay As You Go, Decodo is a great deal for hustlers and fast-growing companies.

Log in to your Decodo dashboard, find residential proxies by choosing Residential under the Residential Proxies column on the left panel, and select a plan that suits your needs. Then, follow these steps:

1. Open the Proxy setup tab.
2. Configure the parameters: set your authentication method, location, session type, and protocol. 
3. Select the number of proxy endpoints you want to generate (default – 10). 
4. Copy the endpoints by clicking the Copy button.

1. Open the settings.py file in your project folder.
2. Edit the file by adding the following properties at the bottom:

DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'yourprojectname.decodo_auth.ProxyMiddleware': 100,
}

SMARTPROXY_USER = 'username'## Decodo username
SMARTPROXY_PASSWORD = 'password'## Decodo password
SMARTPROXY_ENDPOINT = 'gate.decodo.com'## Endpoint you'd like to use
SMARTPROXY_PORT = '7000'## Port of the endpoint you're using

3. Change the yourprojectname in the above to the name of your project folder.

4. Enter the Decodo credentials and endpoint information you’ve received from the dashboard.

5. Your web crawlers will now be going through Decodo proxies.