Someday, you will be blocked by a firewall while trying crawling or accessing some website. The reason is they block your IP address from accessing the server.

One solution is using a web proxy (http proxy, socks4 or socks5) to bypass the firewall, by adding the middle-man server between you and target. It's a bit unsecured but you could use for https site only.

Some HTTP Proxy supports https will stream TLS data from target to you (so don't worry about proxy server can read you data). Btw, it only knows which domain and IP address you're connecting.

To find a free proxy from the internet, try this service : https://gimmeproxy.com/

It provides a cool API to fetch new proxy from its database.

Example this endpoint will return JSON response including proxy anonymous, supports HTTPS, from Japan and minimum speed more than 100KB

http://gimmeproxy.com/api/getProxy?anonymityLevel=1&supportsHttps=false&country=JP&minSpeed=100 

In case you need more requests per day, try a subscription (cancelable and refundable). I tried last days, and really like their service (although I cancelled subscription b/c I don't need proxy anymore).

Break the rules ! ;)

Last month, I built a CI solution for our project and adding a conflict detector to our build commands. This script runned so slow because it will check all application files (and our application codebase has many of css, js files).

This was the script

#!/bin/bash

grep -rli --exclude=conflict_detector.sh --exclude-dir={.git,vendor,node_modules} "<<<<<<< HEAD" .

if [ $? -eq 0 ]; then
    exit 1
else
    exit 0
fi

Today, I think why don't we just check recently updated files (in the latest commit) ??? Then I have this new script

#!/bin/bash

# New way :D
CHANGED_FILES=$(git log --pretty=format: --name-only HEAD^..HEAD | sort | uniq)

for f in $CHANGED_FILES
do
    if grep --exclude=conflict_detector.sh -q "<<<<<<< HEAD" $f
    then
        exit 1
    fi
done

exit 0

conflict_detector.sh is the filename of script, we exclude it from check to make sure changing this file doesn't make it failed.

You can use this approach to check linter, coding standard or run preprocessor ;)

Result (in my context) :

• Old script : 12 seconds
• New script : ~ 50ms (200 times faster)

be Automated, be Fast, but be Careful !

We can get our external IP address by following ways :

1. Call http request : curl http://wtfismyip.com/text or curl http://ifconfig.me/ip
2. Lookup A record for hostname nslookup myip.opendns.com resolver1.opendns.com (this only works when you use resolver of OpenDNS)

Bonus : curl https://v6.ident.me/ for IPv6

We can improve our team workflow by defining some git hooks that trigger on specified events.
You can read all events and their usecases here : https://www.digitalocean.com/community/tutorials/how-to-use-git-hooks-to-automate-development-and-deployment-tasks

This is what I implemented to my today-i-learned repo. I used pre-commit to update Table of Contents in the README.md file, so every content in my repo will be updated on Github repo page.

$ ln pre-commit .git/hooks/pre-commit

pre-commit file :

#!/bin/sh

echo 'Running pre-commit hook' 

python til_update_readme.py
git add README.md

So it will run a Python script that update new TOC and then add the file to git.

Automation ! Automation ! AND .... Automation !!! 🤖

Original MySQL docker image uses a script to generate ssl certificates for service. Sometime we don't really need it (connect via a docker network link or need a fast enough database service to build a automated test).

We can reduce init time by removing the script from original Docker image

FROM mysql:5.7

# Remove mysql_ssl_rsa_setup to ignore setup SSL certs
RUN rm -f /usr/bin/mysql_ssl_rsa_setup

FAST as a FEATURE !!! 🚀

watch to a good command to run a command every N seconds.

And like its name, means you can watch something, its output changes with flag -d

It's a great tool to help you learn a new language without hitting compile and run everytime you save a file.

$ watch -n 1 -d go run learn.go

This command will compile and run learn.go every 1 second

More flags :

• -t : no title
• -b : beep on non-zero exit code
• -e : stop loop on error and exit on a keypress
• -g : exit on change
• -c : support colors
• -h : you know ! ;)

When doing a CI/CD testing, you would need to connect a external service (RDBMS, HTTP server or generic TCP server service). So you need waiting the service before running your test app.

One way to do right waiting instead of sleep for a specified time is using netcat tool

$ while ! echo -e '\x04' | nc [service_host] [service_port]; do sleep 1; done;

Examples

• MySQL service on port 3306

$ while ! echo -e '\x04' | nc 127.0.0.1 3306; do sleep 1; done;
$ ./run_test.sh

Explanation :

echo -e '\x04' will send an EOT (End Of Transmission) to the TCP every second to check if it's ready !

Let's say you have an indexes on 2 columns (A, B) of the table (X). So this is three use cases happen :

1. You query data based on both of 2 columns => Indexes will be considered
2. You query data based on (A) => Indexes will be considered
3. You query data based on (B) => Indexes will be ignored because database indexes your data by B-tree algo. So it can't search node via a B => If you want, just create another indexes on B column

I said will be considered because it depends on your query and your data (query optimizer will decide it !)

You can use netcat or nc as a debugging TCP tool. It can be a TCP sender and receiver with a short session (auto close when connection is closed)

Examples :

Scan ports

$ nc -zv 127.0.0.1 20-80

Check redis status

$ echo 'info' | nc 127.0.0.1 6379

Retrieve http response

$ printf "GET /xinchao HTTP/1.1\r\n\r\n" | nc 127.0.0.1 8000 | tee xinchao.txt

Change to IPv6 : nc -6

Want more ??

$ nc -h

I use python3 (3.4+) to create a bash function to help me start quickly a simple http server on specified port

function server() {
  local port="${1:-8000}" # Getting port number
  google-chrome "http://127.0.0.1:$port" # Open URL in browser, could change to firefox --new-tab "http://127.0.0.1:$port"
  python3 -m http.server $port --bind 127.0.0.1
}

When you want to know how long does it take to run a process, just use time command as a prefix

$ time my_program arg1 arg2
real        0m0.003s
user        0m0.000s
sys         0m0.004s

• real : wall clock time, mean time to start to finish your process
• user : CPUs-time outside the kernel
• sys : CPUs-time within the kernel

real+sys result is total multi CPUs time (so if you have a multi core CPUs, it is often bigger than real)

When a email send through an SMTP (with authentication), every SMTP server is a hop in mail routing. So it will transfer to localmail or forward the email to next hop (shortest distance via DNS MX record).

And standard port of SMTP is 25 (unsecured, but can upgrade to TLS via STARTTLS command).

$ nslookup -type=mx gmail.com 8.8.8.8
Server:         8.8.8.8       
Address:        8.8.8.8#53    

Non-authoritative answer:     
gmail.com       mail exchanger = 20 alt2.gmail-smtp-in.l.google.com.                                                    
gmail.com       mail exchanger = 5 gmail-smtp-in.l.google.com.                                                          
gmail.com       mail exchanger = 30 alt3.gmail-smtp-in.l.google.com.                                                    
gmail.com       mail exchanger = 10 alt1.gmail-smtp-in.l.google.com.                                                    
gmail.com       mail exchanger = 40 alt4.gmail-smtp-in.l.google.com.                                                    

Authoritative answers can be found from:

So shortest SMTP of gmail.com domain is gmail-smtp-in.l.google.com

$ telnet gmail-smtp-in.l.google.com 25

When you try to send an Enter keyboard to linux kernel, it looks like nothing happens.

This is because you only send a key press (KEY DOWN) but don't send an key release (KEY UP) event after that.

Thank Hiro Ishii for teaching me this

set -x will print all running commands in your bash script

So I dove in and look for all set options of BASH.

And this is what I got , http://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html

Creating function is a good way to refactor your bash script. But BASH doesn't support returning a value in true way, so it makes a bit of challenge to handle that.

You can use this trick

hello() {
	echo "Hello $1"
}

hw=$(hello "KhanhIceTea")
echo $hw

But what if you want to echo log message in hello function, it will be merged to returned value.

hello() {
	echo "System is weaking up and brush its teeth :))"
	echo "Hello $1"
}

hw=$(hello "KhanhIceTea")
echo "This is returned value of hello function :"
echo $hw

This is a how to resolve it, forwarding log message to stderr instead of stdout by default

hello() {
	echo "System is weaking up and brush its teeth :))" >&2
	echo "Hello $1"
}

hw=$(hello "KhanhIceTea")
echo "This is returned value of hello function :"
echo $hw

Where there is a shell, there is a way !

Setting a flag set -e to bash script will let the script exit on first error occurs, so if you want to ignore a command just adding || true to suffix

set -e

errorCmd $1 || true
echo "Run here !"

And opposite of set -e is set +e, haha of course !

set +e

errorCmd $1
echo "Still run here !"

To specify a list of compressed files when using zip cli tool, you could use -@ [file_list] flag. And file_list is a file contains list of compressed file (new line separated)

Example

$ zip changed.zip -@ changed_files.txt

Or use stdin pipe

$ find . -mmin -60 -print | zip changed_1_hour_ago -@

This will zip all changed files 1 hour ago

Last day I checked system logs and got a lot of warning messages mentioned that my server has been attack via Brute-force. So I decided to blocked some countries from connecting to attacked ports (21, 25). They are China, Russia and US.

This site provides a list of IP blocks of specified country

http://www.ipdeny.com/ipblocks/

The TCP FIN timeout belays the amount of time a port must be inactive before it can reused for another connection. The default is often 60 seconds, but can normally be safely reduced to 30 or even 15 seconds:

net.ipv4.tcp_fin_timeout = 15

Ref : https://www.linode.com/docs/web-servers/nginx/configure-nginx-for-optimized-performance

In Linux, you can prevent a user from login by locking it.

Lock

$ sudo passwd -l [user]

Unlock

$ sudo passwd -u [user]

openssl uses strong prime (which is useless for security but requires an awful lot more computational effort. A "strong prime" is a prime p such that (p-1)/2 is also prime). So it will be faster if we add option -dsaparam to the command

$ openssl dhparam -dsaparam -out /etc/ssl/private/dhparam.pem 4096

Ref : https://security.stackexchange.com/a/95184

Grep is a greate tool for searching a string in files.

Syntax

$ grep -nr '[string]' [folder]

If you want to show surrounding lines the result, add flag -C [number] to the command

$ grep -nr -C 3 'hello' src

You can skip to a task by its name by adding parameter --start-at

$ ansible-playbook playbook.yml --start-at="[your task name]"

This tool is written in Python with super cool features (auto-complete and colors).

Worth a shot !

Install

$ pip install mycli

Usage

$ mycli -h 127.0.0.1 -P 3306 -u root

Screencast

CentOS with SELinux enabled by default will block any http proxy connection. So you have to enable this permission.

Temporary enable

$ /usr/sbin/setsebool httpd_can_network_connect 1

Permanent enable

$ /usr/sbin/setsebool -P httpd_can_network_connect 1

SSH Tunnel is a fast way to transfer traffic through unsafe internet today. It would be used in MySQL connect, FTP connect or HTTP connect, ...

Syntax :

$ ssh -L [local_port]:[remote_endpoint]:[remote_port] [ssh_user]:[ssh_ip]

Example :

Lets say you have a EC2 instance (123.45.67.89) and remote DB instance (98.76.54.32) listening port 3306

$ ssh -L 3307:98.76.54.32:3306 root@123.45.67.89

Testing ssh tunnel

$ telnet 127.0.0.1 3307
$ # or
$ mysql -h 127.0.0.1 -P 3307 -u root -p

Run this code

for (var i=1; i<=5; i++) {
	setTimeout( function timer(){
		console.log( i );
	}, i*1000 );
}

What you expected

1
2
3
4
5

But, result is

6
6
6
6
6

Solution is

for (var i = 1; i <= 5; i++) {
    setTimeout((function timer(j) {
        return function() {
            console.log(j);
        }
    })(i), i * 1000);
}

or

for (var i=1; i<=5; i++) {
	(function(j){
		setTimeout( function timer(){
			console.log( j );
		}, j*1000 );
	})(i);
}

JS code will be slower if engine detects any of 'eval' function or 'with' block b/c compiler stop optimizing the code

Edit file /etc/default/keyboard and set

XKBOPTIONS="ctrl:nocaps"

Then, logout and log in again to impact

When you post new content to your website, the fastest way is ping search engines to notify them. After that, they will try to crawl and index your page.

One way to ping search engines is using XMLRPC ping

This is a example XMLRPC request (HTTP POST request with xml body)

Request

> POST /ping/RPC2 HTTP/1.1
> Host: blogsearch.google.com
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/xml
> Content-Length: 239
> 
<?xml version="1.0" encoding="UTF-8"?>
<methodCall>
   <methodName>weblogUpdates.extendedPing</methodName>
   <params>
      <param>
         <value>Page Title</value>
      </param>
      <param>
         <value>http://example.com/helloworld.html</value>
      </param>
   </params>
</methodCall>

Response

< HTTP/1.1 200 OK
< Content-Type: text/xml; charset=ISO-8859-1
< X-Content-Type-Options: nosniff
< Date: Tue, 08 Aug 2017 05:04:01 GMT
< Server: psfe
< Cache-Control: private
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< 
<?xml version="1.0"?>
<methodResponse><params>
  <param><value><struct>
    <member>
      <name>flerror</name><value><boolean>0</boolean></value>
    </member>
    <member>
      <name>message</name><value>Thanks for the ping.</value>
    </member>
  </struct></value></param>
</params></methodResponse>

Popular XML Servers

http://blogsearch.google.com/ping/RPC2
http://api.moreover.com/ping
http://bblog.com/ping.php
http://bitacoras.net/ping
http://blog.goo.ne.jp/XMLRPC
http://blogmatcher.com/u.php
http://coreblog.org/ping/
http://mod-pubsub.org/kn_apps/blogchatt
http://www.lasermemory.com/lsrpc/
http://ping.amagle.com/
http://ping.cocolog-nifty.com/xmlrpc
http://ping.exblog.jp/xmlrpc
http://ping.feedburner.com
http://ping.myblog.jp
http://ping.rootblog.com/rpc.php
http://ping.syndic8.com/xmlrpc.php
http://ping.weblogalot.com/rpc.php
http://pingoat.com/goat/RPC2
http://rcs.datashed.net/RPC2/
http://rpc.blogrolling.com/pinger/
http://rpc.pingomatic.com
http://rpc.technorati.com/rpc/ping
http://rpc.weblogs.com/RPC2
http://www.blogpeople.net/servlet/weblogUpdates
http://www.blogroots.com/tb_populi.blog?id=1
http://www.blogshares.com/rpc.php
http://www.blogsnow.com/ping
http://www.blogstreet.com/xrbin/xmlrpc.cgi
http://xping.pubsub.com/ping/