Someday, you will be blocked by a firewall while trying crawling or accessing some website. The reason is they block your IP address from accessing the server.
One solution is using a web proxy (http proxy, socks4 or socks5) to bypass the firewall, by adding the middle-man server between you and target. It’s a bit unsecured but you could use for https site only.
Some HTTP Proxy supports https will stream TLS data from target to you (so don’t worry about proxy server can read you data). Btw, it only knows which domain and IP address you’re connecting.
In case you need more requests per day, try a subscription (cancelable and refundable). I tried last days, and really like their service (although I cancelled subscription b/c I don’t need proxy anymore).
If you are using Cloudflare as a proxied web server, it will provide many benefits about performance (assets caching, prevent DDOS and cheap CDN). But sometimes, you will face to this error “522 Connection Time out”.
Server Software: Werkzeug/0.12.1 Server Hostname: 127.0.0.1 Server Port: 5000
Document Path: / Document Length: 10 bytes
Concurrency Level: 5 Time taken for tests: 0.537 seconds Complete requests: 1000 Failed requests: 683 (Connect: 0, Receive: 0, Length: 683, Exceptions: 0) Total transferred: 164620 bytes HTML transferred: 9965 bytes Requests per second: 1862.55 [#/sec] (mean) Time per request: 2.684 [ms] (mean) Time per request: 0.537 [ms] (mean, across all concurrent requests) Transfer rate: 299.43 [Kbytes/sec] received
Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 1 3 0.7 2 11 Waiting: 1 2 0.6 2 11 Total: 1 3 0.7 3 11 WARNING: The median and mean for the processing time are not within a normal deviation These results are probably not that reliable.
Percentage of the requests served within a certain time (ms) 50% 3 66% 3 75% 3 80% 3 90% 3 95% 3 98% 5 99% 6 100% 11 (longest request)
In this example, first response content-length is 10 (“hello” x 2), so every responses has content length is 5 or 15, will be assumed a failed response.