#TIL : HSTS rule in browser

HTTP Strict Transport Security (HSTS) is a web security policy mechanism which helps to protect websites against protocol downgrade attacks and cookie hijacking.

Enabling HSTS on your web will make your browser validate every SSL issues more strictly :

  • User can not visit http version on browser
  • User can not add SSL exception for the domain to ignore the warning. (when SSL cert expire or invalid common name)

Note : You can manually remove a domain from HSTS in Chrome by accessing this page URL chrome://net-internals/#hsts

So remember to add HSTS to your website !

#TIL : Using web proxy to bypass firewalls

Someday, you will be blocked by a firewall while trying crawling or accessing some website. The reason is they block your IP address from accessing the server.

One solution is using a web proxy (http proxy, socks4 or socks5) to bypass the firewall, by adding the middle-man server between you and target. It’s a bit unsecured but you could use for https site only.

Some HTTP Proxy supports https will stream TLS data from target to you (so don’t worry about proxy server can read you data). Btw, it only knows which domain and IP address you’re connecting.

To find a free proxy from the internet, try this service : https://gimmeproxy.com/

It provides a cool API to fetch new proxy from its database.

Example this endpoint will return JSON response including proxy anonymous, supports HTTPS, from Japan and minimum speed more than 100KB

1
http://gimmeproxy.com/api/getProxy?anonymityLevel=1&supportsHttps=false&country=JP&minSpeed=100

In case you need more requests per day, try a subscription (cancelable and refundable). I tried last days, and really like their service (although I cancelled subscription b/c I don’t need proxy anymore).

Break the rules ! ;)

#TIL : Enable reverse proxy in CentOS

CentOS with SELinux enabled by default will block any http proxy connection. So you have to enable this permission.

Temporary enable

1
$ /usr/sbin/setsebool httpd_can_network_connect 1

Permanent enable

1
$ /usr/sbin/setsebool -P httpd_can_network_connect 1

#TIL : Ping Google to crawl updated content

When you post new content to your website, the fastest way is ping search engines to notify them. After that, they will try to crawl and index your page.

One way to ping search engines is using XMLRPC ping

This is a example XMLRPC request (HTTP POST request with xml body)

Request

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
> POST /ping/RPC2 HTTP/1.1
> Host: blogsearch.google.com
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/xml
> Content-Length: 239
>
<?xml version="1.0" encoding="UTF-8"?>
<methodCall>
<methodName>weblogUpdates.extendedPing</methodName>
<params>
<param>
<value>Page Title</value>
</param>
<param>
<value>http://example.com/helloworld.html</value>
</param>
</params>
</methodCall>

Response

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
< HTTP/1.1 200 OK
< Content-Type: text/xml; charset=ISO-8859-1
< X-Content-Type-Options: nosniff
< Date: Tue, 08 Aug 2017 05:04:01 GMT
< Server: psfe
< Cache-Control: private
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
<
<?xml version="1.0"?>
<methodResponse><params>
<param><value><struct>
<member>
<name>flerror</name><value><boolean>0</boolean></value>
</member>
<member>
<name>message</name><value>Thanks for the ping.</value>
</member>
</struct></value></param>
</params></methodResponse>

Popular XML Servers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
http://blogsearch.google.com/ping/RPC2
http://api.moreover.com/ping
http://bblog.com/ping.php
http://bitacoras.net/ping
http://blog.goo.ne.jp/XMLRPC
http://blogmatcher.com/u.php
http://coreblog.org/ping/
http://mod-pubsub.org/kn_apps/blogchatt
http://www.lasermemory.com/lsrpc/
http://ping.amagle.com/
http://ping.cocolog-nifty.com/xmlrpc
http://ping.exblog.jp/xmlrpc
http://ping.feedburner.com
http://ping.myblog.jp
http://ping.rootblog.com/rpc.php
http://ping.syndic8.com/xmlrpc.php
http://ping.weblogalot.com/rpc.php
http://pingoat.com/goat/RPC2
http://rcs.datashed.net/RPC2/
http://rpc.blogrolling.com/pinger/
http://rpc.pingomatic.com
http://rpc.technorati.com/rpc/ping
http://rpc.weblogs.com/RPC2
http://www.blogpeople.net/servlet/weblogUpdates
http://www.blogroots.com/tb_populi.blog?id=1
http://www.blogshares.com/rpc.php
http://www.blogsnow.com/ping
http://www.blogstreet.com/xrbin/xmlrpc.cgi
http://xping.pubsub.com/ping/

#TIL : Cloudflare Error 522 Connection Time out

If you are using Cloudflare as a proxied web server, it will provide many benefits about performance (assets caching, prevent DDOS and cheap CDN). But sometimes, you will face to this error “522 Connection Time out”.

The problems caused by :

  • Networking (CF can’t touch origin server : Firewall blocking, Network Layer #1,#2,#3 issue)
  • Timeout (origin server process too long than 90 seconds)
  • Empty or invalid response from origin server
  • No or big HTTP headers (> 8Kb)
  • Failed TCP handshake

Ref:

#TIL : ab failed responses

When benchmarking a HTTP application server using ab tool, you shouldn’t only care about how many requests per second, but percentage of Success responses.

A notice that you must have the same content-length in responses, because ab tool will assume response having different content-length from Document Length (in ab result) is failed response.

Example

Webserver using Flask

1
2
3
4
5
6
7
8
9
10
from flask import Flask
from random import randint
app = Flask(__name__)

@app.route("/")
def hello():
return "Hello" * randint(1,3)

if __name__ == "__main__":
app.run()

Benchmark using ab

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
$ ab -n 1000 -c 5 http://127.0.0.1:5000/

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: Werkzeug/0.12.1
Server Hostname: 127.0.0.1
Server Port: 5000

Document Path: /
Document Length: 10 bytes

Concurrency Level: 5
Time taken for tests: 0.537 seconds
Complete requests: 1000
Failed requests: 683
(Connect: 0, Receive: 0, Length: 683, Exceptions: 0)
Total transferred: 164620 bytes
HTML transferred: 9965 bytes
Requests per second: 1862.55 [#/sec] (mean)
Time per request: 2.684 [ms] (mean)
Time per request: 0.537 [ms] (mean, across all concurrent requests)
Transfer rate: 299.43 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 3 0.7 2 11
Waiting: 1 2 0.6 2 11
Total: 1 3 0.7 3 11
WARNING: The median and mean for the processing time are not within a normal deviation
These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
50% 3
66% 3
75% 3
80% 3
90% 3
95% 3
98% 5
99% 6
100% 11 (longest request)

In this example, first response content-length is 10 (“hello” x 2), so every responses has content length is 5 or 15, will be assumed a failed response.

#TIL : Base 64 encode and decode builtin tool

Browsers have helpers function to encode and decode base64 :

  • btoa : base64 encode
  • atob : base64 decode
1
2
3
4
5
> btoa('Hello world')
"SGVsbG8gV29ybGQgIQ=="

> atob('SW4gR29kIFdlIFRydXN0ICE=')
"In God We Trust !"