This is my config to use 4 spaces instead tab

filetype plugin indent on
set tabstop=4
set shiftwidth=4
set expandtab

To convert existing file from tabs to spaces, use this command

:%retab

In Redis, you can use transaction-style (mean queue commands then flush it once). It will improve performance in many case where latency or networking is slow.

> SET hoge 2
OK
> MULTI
OK
> INCR foo
QUEUED
> INCR hoge
QUEUED
> EXEC
1) (integer) 1
2) (integer) 1

MULTI command is begin transaction and EXEC command is commit transaction

The result will be returned in order your command queue list.

Like NginX, Varnish has a syntax checker function that helps us test the syntactic correctness.

$ varnishd -C -f [vcl file path]

Varnish will compile the file and output the result to stdout. If something goes wrong, it will throw a message like

> Message from VCC-compiler:
> Expected an action, 'if', '{' or '}'
> ('input' Line 74 Pos 6)
>     vcl_hash(req.http.Cookie);
> -----########------------------
>
> Running VCC-compiler failed, exit 1

Bookmarking a checkpoint will help you get back to it intermidately. Ex: your have to find some text to replace something but want to return back current position.

Set a mark

• [NORMAL MODE] , type m then follow by a letter from a-z (lowercase is filescope, uppercase for global scope - vim scope)

Go to a mark

• [NORMAL MODE] , type backstick ` then follow by the letter your marked above.

List all current marks

• [NORMAL MODE], :marks
• It shows all marks included special ones :

TIPS : Can use it as a motion with change, delete or yank

Last days, I developed a EFK stack to centralize my system logging. I really like the concepts of FluentD, it's better than original stack ELK of elastic company.

So I need to learn basics about Elasticsearch and Kibana

This is what I learned :

# Get all documents from elasticsearch node
GET _search
{
  "query": {
    "match_all": {}
  }
}

# Check nodes statistics
GET /_nodes/stats

# Check health of cluster (I don't know why it is always yello status)
GET _cluster/health

# Get list of indices (indexes)
GET /_cat/indices?v

# Delete a index (with its data) with name
DELETE /[index-name]

Logging and Monitoring are important factor for system admin. Checking the log will help you have a closer look into the issue. One tool could help you will handy features is journalctl.

Here are simple options :

• -f : follow the log (tailf)
• -u [service] : filter to show only [service] logs
• --since=[date] : Show entries not older than the specified date
• --until=[date] : Show entries not newer than the specified date

Example :

$ sudo journalctl -u nginx.service
$ sudo journalctl -u nginx.service --since yesterday
$ sudo journalctl -u nginx.service --since "2018-01-01" --until today

Ansible supports pattern to select and filter running hosts from all hosts. Here is some common pattern

• * : wildcard, standalone mean all
• group1,group2 : run hosts belong to group1 or group2
• *.web : run hosts belongs to group matches pattern. Ex: backend.web, frontend.web
• all:!abc : run all hosts exclude hosts belongs to group abc

Infrastructure by code ;)

Allow unmetrered connections on networking is so risky. Attacker can use the brute-force attacks to comprosise your service (or simple DOS).

Linux has a cool firewall to hanlde this, via ip-tables. But it's so complicated to remember all the rule and syntax. That's why UFW was born to save us. :D

You can use simple command to manage your firewall

$ ufw default deny incoming # deny any incoming port, should be run before allow any port
$ ufw default allow outgoing # allow any outgoing port
$ ufw allow 80 # allow port 80
$ ufw deny 53/udp # allow udp protocol to port 53
$ ufw disable # disable firewall
$ ufw enable # enable firewall
$ ufw status # check all the rules
$ ufw delete [num] # delete the rule by its order in status result
$ ufw reload # reload all rule
$ ufw limit ssh/tcp # finnaly, limit ssh (port 22 tcp), deny connections if an IP address has attempted to initiate 6 or more connections in the last 30 seconds

I know IPv6 will be future for networking, but at this moment "It's suck !" :(

Some service will be failed when trying to connect IPv6 destination :

• apt package manager
• smtp
• curl

So I decided to disable IPv6 on every production server.

$ echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
$ echo "net.ipv6.conf.default.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
$ echo "net.ipv6.conf.lo.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
$ 
$ sudo sysctl -p

I will re-enable it when everything works perfectly !

TLDR;

Biển xanh lại mặn bởi vì đá ở mặt đất cọ xát với mưa trên trời (chứ không phải cá nó đái 😂 )

Read more : https://oceanservice.noaa.gov/facts/whysalty.html

After setting value of an input via val method, we should call the change chaining method to trigger the onChange event of element.

$('#selectCity').change(function() {
	console.log($(this).val());
});

$('#selectCity').val('HaNoi'); // No trigger

$('#selectCity').val('HoChiMinh').change(); // Fire trigger

TLDR;

Biển xanh lại mặn bởi vì đá ở mặt đất cọ xát với mưa trên trời (chứ không phải cá nó đái 😂 )

Read more : https://oceanservice.noaa.gov/facts/whysalty.html

Using Object.defineProperty helper function as I wrote 3 days ago. We could track the changes of cookie on webpage.

// Based on Vlad Shevchenko's script at https://stackoverflow.com/a/36826049

var cookieSetterOrig = document.__lookupSetter__("cookie"); // get origin setter function
var cookieGetterOrig = document.__lookupGetter__("cookie"); // get origin getter function
Object.defineProperty(document, "cookie", {
    get: function () {
	console.trace();
        return cookieGetterOrig.apply(document);
    },
    set: function () {
	console.log(arguments);
	console.trace();
        return cookieSetterOrig.apply(document, arguments);
    },
    configurable: true
});

Notice : This code only works if cookie is changed by javascript, not http header request !

Sometimes you are blocked from request a cross-origin resource. Instead of adding our domain to allowed list of them, we can use another way to retrieve data from their API by using JSONP (in case they support it).

The mechanism of JSONP is simple, instead of returning a JSON data. It will return a javascript text with passing your data into a function, whose name is declared in query string. So you just add a new script element with the URL and waiting the callback.

Example :

function callMeBaby(data) {
	console.log(data);
}

var s = document.createElement("script");
s.type = "text/javascript";
s.src = "https://freegeoip.net/json/?callback=callMeBaby";
document.head.appendChild(s);

or using jQuery (hide magic)

$.ajax({
    url: "https://freegeoip.net/json/",
    jsonp: "callback",
    dataType: "jsonp",
    success: function( data ) {
        console.log( data );
    }
});

Sometimes, we want to define a property of an advanced object (has setter and getter function).

Now, we could use this helper function Object.defineProperty to define property of an object in a cool way.

Example :

const foo = {};

Object.defineProperty(a, 'bar', {
	value: 'hogehoge',
	writable: false,
});

console.log(foo.bar); // 'hogehoge'
foo.bar = 'foo bar'; // throw an error in strict mode
console.log(foo.bar); // still be 'hogehoge'

Modifying setter, getter function

// Get callstack which function is getting or setting cookie value
Object.defineProperty(document, 'cookie', {
	get: function() {
		console.log('get !');
		console.trace();
	},
	set: function(val) {
		console.log('set = ', val);
		console.trace();
	},
}); 

Browsers provide an useful function help you debug easier than using simple console.log function.

That is console.trace, which prints a stack trace to called function.

Example :

function foo() {
	var a = 1;
	bar(a);
}
function bar(x) {
	console.log(x);
	console.trace();
}

foo();

When you check your MySQL process list via command show processlist;, it will show you a useful table which provide all current connection details.

"Sleep" state connections are most connection pointer waiting for the timeout to terminate. Then they still count as a connection. (Can cause MySQL connection limit error, which default equal 150 connections)

So next time, remember to close your connection before terminating your app.

Every connection counts ;)

HTTP Strict Transport Security (HSTS) is a web security policy mechanism which helps to protect websites against protocol downgrade attacks and cookie hijacking.

Enabling HSTS on your web will make your browser validate every SSL issues more strictly :

• User can not visit http version on browser
• User can not add SSL exception for the domain to ignore the warning. (when SSL cert expire or invalid common name)

Note : You can manually remove a domain from HSTS in Chrome by accessing this page URL chrome://net-internals/#hsts

So remember to add HSTS to your website !

You have a mobile app for both platforms iOS and Android, each platform has different download URL. But your user doesn't know which platform he using. Clicking wrong URL will lead to user bounce-rate.

Solution is making only 1 URL to download your app, which can redirect to right place depends on using platform. So how we achieve this ??

The key of problem is detecting user platform, which can be done by extracting the User-Agent header from http request.

This is how I implement using Caddy web server, you can do same thing in Apache or NGINX

app.yourcompany.com:443 {
    timeouts 1m

    redir 302 {
        if_op or
        if {>User-agent} has iPhone
        if {>User-agent} has iPod
        if {>User-agent} has iPad
	/ [apple-store-url]
    }

    redir 302 {
        if {>User-agent} has Android
	/ [google-play-url]
    }

    redir 302 {
        if {>User-agent} not_has iPhone
        if {>User-agent} not_has iPod
        if {>User-agent} not_has iPad
        if {>User-agent} not_has Android
        / [your-landing-page-which-user-visit-on-desktop-device]
    }
}

Then we get cool and easy to remember link, right ?

Someday, you will be blocked by a firewall while trying crawling or accessing some website. The reason is they block your IP address from accessing the server.

One solution is using a web proxy (http proxy, socks4 or socks5) to bypass the firewall, by adding the middle-man server between you and target. It's a bit unsecured but you could use for https site only.

Some HTTP Proxy supports https will stream TLS data from target to you (so don't worry about proxy server can read you data). Btw, it only knows which domain and IP address you're connecting.

To find a free proxy from the internet, try this service : https://gimmeproxy.com/

It provides a cool API to fetch new proxy from its database.

Example this endpoint will return JSON response including proxy anonymous, supports HTTPS, from Japan and minimum speed more than 100KB

http://gimmeproxy.com/api/getProxy?anonymityLevel=1&supportsHttps=false&country=JP&minSpeed=100 

In case you need more requests per day, try a subscription (cancelable and refundable). I tried last days, and really like their service (although I cancelled subscription b/c I don't need proxy anymore).

Break the rules ! ;)

Last month, I built a CI solution for our project and adding a conflict detector to our build commands. This script runned so slow because it will check all application files (and our application codebase has many of css, js files).

This was the script

#!/bin/bash

grep -rli --exclude=conflict_detector.sh --exclude-dir={.git,vendor,node_modules} "<<<<<<< HEAD" .

if [ $? -eq 0 ]; then
    exit 1
else
    exit 0
fi

Today, I think why don't we just check recently updated files (in the latest commit) ??? Then I have this new script

#!/bin/bash

# New way :D
CHANGED_FILES=$(git log --pretty=format: --name-only HEAD^..HEAD | sort | uniq)

for f in $CHANGED_FILES
do
    if grep --exclude=conflict_detector.sh -q "<<<<<<< HEAD" $f
    then
        exit 1
    fi
done

exit 0

conflict_detector.sh is the filename of script, we exclude it from check to make sure changing this file doesn't make it failed.

You can use this approach to check linter, coding standard or run preprocessor ;)

Result (in my context) :

• Old script : 12 seconds
• New script : ~ 50ms (200 times faster)

be Automated, be Fast, but be Careful !

We can get our external IP address by following ways :

1. Call http request : curl http://wtfismyip.com/text or curl http://ifconfig.me/ip
2. Lookup A record for hostname nslookup myip.opendns.com resolver1.opendns.com (this only works when you use resolver of OpenDNS)

Bonus : curl https://v6.ident.me/ for IPv6

We can improve our team workflow by defining some git hooks that trigger on specified events.
You can read all events and their usecases here : https://www.digitalocean.com/community/tutorials/how-to-use-git-hooks-to-automate-development-and-deployment-tasks

This is what I implemented to my today-i-learned repo. I used pre-commit to update Table of Contents in the README.md file, so every content in my repo will be updated on Github repo page.

$ ln pre-commit .git/hooks/pre-commit

pre-commit file :

#!/bin/sh

echo 'Running pre-commit hook' 

python til_update_readme.py
git add README.md

So it will run a Python script that update new TOC and then add the file to git.

Automation ! Automation ! AND .... Automation !!! 🤖

Original MySQL docker image uses a script to generate ssl certificates for service. Sometime we don't really need it (connect via a docker network link or need a fast enough database service to build a automated test).

We can reduce init time by removing the script from original Docker image

FROM mysql:5.7

# Remove mysql_ssl_rsa_setup to ignore setup SSL certs
RUN rm -f /usr/bin/mysql_ssl_rsa_setup

FAST as a FEATURE !!! 🚀

watch to a good command to run a command every N seconds.

And like its name, means you can watch something, its output changes with flag -d

It's a great tool to help you learn a new language without hitting compile and run everytime you save a file.

$ watch -n 1 -d go run learn.go

This command will compile and run learn.go every 1 second

More flags :

• -t : no title
• -b : beep on non-zero exit code
• -e : stop loop on error and exit on a keypress
• -g : exit on change
• -c : support colors
• -h : you know ! ;)

When doing a CI/CD testing, you would need to connect a external service (RDBMS, HTTP server or generic TCP server service). So you need waiting the service before running your test app.

One way to do right waiting instead of sleep for a specified time is using netcat tool

$ while ! echo -e '\x04' | nc [service_host] [service_port]; do sleep 1; done;

Examples

• MySQL service on port 3306

$ while ! echo -e '\x04' | nc 127.0.0.1 3306; do sleep 1; done;
$ ./run_test.sh

Explanation :

echo -e '\x04' will send an EOT (End Of Transmission) to the TCP every second to check if it's ready !

Let's say you have an indexes on 2 columns (A, B) of the table (X). So this is three use cases happen :

1. You query data based on both of 2 columns => Indexes will be considered
2. You query data based on (A) => Indexes will be considered
3. You query data based on (B) => Indexes will be ignored because database indexes your data by B-tree algo. So it can't search node via a B => If you want, just create another indexes on B column

I said will be considered because it depends on your query and your data (query optimizer will decide it !)

You can use netcat or nc as a debugging TCP tool. It can be a TCP sender and receiver with a short session (auto close when connection is closed)

Examples :

Scan ports

$ nc -zv 127.0.0.1 20-80

Check redis status

$ echo 'info' | nc 127.0.0.1 6379

Retrieve http response

$ printf "GET /xinchao HTTP/1.1\r\n\r\n" | nc 127.0.0.1 8000 | tee xinchao.txt

Change to IPv6 : nc -6

Want more ??

$ nc -h

I use python3 (3.4+) to create a bash function to help me start quickly a simple http server on specified port

function server() {
  local port="${1:-8000}" # Getting port number
  google-chrome "http://127.0.0.1:$port" # Open URL in browser, could change to firefox --new-tab "http://127.0.0.1:$port"
  python3 -m http.server $port --bind 127.0.0.1
}

When you want to know how long does it take to run a process, just use time command as a prefix

$ time my_program arg1 arg2
real        0m0.003s
user        0m0.000s
sys         0m0.004s

• real : wall clock time, mean time to start to finish your process
• user : CPUs-time outside the kernel
• sys : CPUs-time within the kernel

real+sys result is total multi CPUs time (so if you have a multi core CPUs, it is often bigger than real)