1. Keep learning forward

    To be updated ...
  2. #TIL : Create tiny chat channel via netcat

    In a network, you can create a tiny chatting channel using netcat. It's lightweight TCP protocol with plain-text transmission, so be carefully on using.

    First, create a channel by picking port number (ex: 7777)

    $ sudo nc -l 0.0.0.0 7777

    Then, tell you friend your IP and channel port. He will use this info to connect the channel

    $ nc 192.168.1.2 7777

    Finnally, start chatting !! Each message will be send when you press [Enter]

    Note: End the session by press Ctrl + D

    netcat chatting

  3. #TIL : Prepend line number to file

    When you want to prepend line number in every line of file, use the -n flag of cat tool.

    Example :

    cat -n a.txt

    Or even from many file

    cat -n a.txt b.txt c.txt

  4. #TIL : Cut file content from line to line

    In case you have a big file which contains a lot of content (2+ GB). And you only need a small part from the file (the part is continuous string from line X to line Y).

    You have many ways to achieve that :

    1. Use vi editor and delete from line 1 to line (X-1) by press [X-1]dd then go to line (Y-X+2) and delete to last line by press dG
    2. Use sed -n '[X][Y]p' [input_file] > [output_file]. Example : sed -n '15,68p' a.sql > b.sql
    3. Use head and tail trick : head -n[Y] [input_file] | tail -n[Y-X+1] > [output_file]

    I personally recommend using the sed way, it's faster and simpler to remember.

  5. #TIL : Create a sequence of numbers

    In the past, every time I want to create a sequence of numbers. I have to use something like MS EXCEL, then copy it and paste to text editor. It's tricky way and slow !

    Now, I can use the handy tool seq to achieve that

    man seq

    SEQ(1)                                          User Commands                                         SEQ(1)            
    
    NAME                          
           seq - print a sequence of numbers                    
    
    SYNOPSIS                      
           seq [OPTION]... LAST   
           seq [OPTION]... FIRST LAST                           
           seq [OPTION]... FIRST INCREMENT LAST                 
    
    DESCRIPTION                   
           Print numbers from FIRST to LAST, in steps of INCREMENT.                                                         
    
           Mandatory arguments to long options are mandatory for short options too.                                         
    
           -f, --format=FORMAT    
                  use printf style floating-point FORMAT        
    
           -s, --separator=STRING 
                  use STRING to separate numbers (default: \n)  
    
           -w, --equal-width      
                  equalize width by padding with leading zeroes
    

    So we have 3 main arguments (same as for loop) :

    • FIRST
    • INCREMENT
    • LAST

    And 3 options :

    • format : you can use string format like This is number %g
    • separetor : default is new line
    • equal width : padding with leading zeroes

    Example :

    $ seq -f"This is number %g" 3 4 20
    This is number 3              
    This is number 7              
    This is number 11             
    This is number 15             
    This is number 19
    $ seq -w -s", " 10
    01, 02, 03, 04, 05, 06, 07, 08, 09, 10
  6. #TIL : Install CA root certificate on iOS device

    Disclaimer : ⚠️ You can do it, but it's at your own risk !

    Sometimes you want to accept a SSL firewall proxy or self-MITM proxy, the important step is installing its CA root certificate to your device. Because iOS apps almost use all https connections (that's new rule).

    This is the way to install and enable custom CA Root cert :

    • Step 1 : encode your certificate to binary-PEM (only need when you try cat [ca-cert] and see ASCII base64 characters)
    openssl x509 -outform der -in [ca-cert] -out [new-ca-cert].crt
    • Step 2 : Transfer the root certificate to your device (can use 1 of 2 methods : uploading cert to public webserver and open link in Safari app; or share certificate file through AirDrop - between 2 Apple devices).

    Tips : use ngrok as a simple tunnel webserver if you don't have AirDrop supported PC.

    • Step 3 : Click Install on install profile screen

    • Step 4 : Enable installed certificate, go to Settings > General > About > Certificate Trust Settings, then switch On your certificate item. (You could disable it when you don't need it)

    ;) Check the web connection !

  7. #TIL : Flush DNS cache on iOS device

    There are 2 simple ways to clear DNS cache on iOS devices :

    1. (FASTER) Just enable the Airplane Mode, wait 10 seconds and disable it
    2. (SLOWER) Reboot the device ! ;) You know this always be classic answer for many questions :D
  8. #TIL : Sending Cookie in AJAX CORs request

    By default, browser will remove the cookie and authorization header from AJAX CORs request. So
    before sending out the request, make sure withCredentials must be true.

    In this case, CORs response must specify which origin is allowed (mean
    no wildcard allowed origin rule).

  9. #TIL : Prevent source hacking from .git directory exposing

    Many web project use Git as source version control tools. So in production
    server, we could expose the hidden .git directory - which contains all most
    infomation about project source code.

    To "rip" a source code from a vulnerable website, we can use this tool : https://github.com/kost/dvcs-ripper#git

    So to prevent this happens, try to deny all http access to hidden files and
    directories (usually starts by . character)

    Example of Nginx config

    location ~ /\. {
        deny all;
    }
    
  10. #TIL : Build lightweight image by using multistage

    Docker is great tool to build a pull-n-run application. But sometimes, your image will be large if you build image from a big base image which has heavy compliling toolbox.

    Ex:

    One-stage build

    FROM golang:1.9.2
    WORKDIR /go/src/github.com/khanhicetea/test/
    COPY . .
    RUN CGO_ENABLED=0 GOOS=linux go build .
    ENTRYPOINT ["/go/src/github.com/khanhicetea/test/test"]
    

    Multi-stage builds

    FROM golang:1.9.2
    WORKDIR /go/src/github.com/khanhicetea/test/
    COPY . .
    RUN CGO_ENABLED=0 GOOS=linux go build .
    
    FROM scratch
    COPY --from=0 /go/src/github.com/khanhicetea/test/test .
    ENTRYPOINT ["/test"]
    

    So final image will only contains /test excutable file. COPY --from=0 means COPY from build has index 0 in Dockerfile.

  11. #TIL : List opening ports or listening UNIX sockets

    In Linux, you can use netstat to list all opening ports and listening UNIX sockets

    $ sudo netstat -npl

    Tip to remember command : network statistics - natual languge processing

    ;)

  12. #TIL : Convert tabs to spaces

    This is my config to use 4 spaces instead tab

    filetype plugin indent on
    set tabstop=4
    set shiftwidth=4
    set expandtab
    

    To convert existing file from tabs to spaces, use this command

    :%retab
    
  13. #TIL : Transaction style in Redis

    In Redis, you can use transaction-style (mean queue commands then flush it once). It will improve performance in many case where latency or networking is slow.

    > SET hoge 2
    OK
    > MULTI
    OK
    > INCR foo
    QUEUED
    > INCR hoge
    QUEUED
    > EXEC
    1) (integer) 1
    2) (integer) 1
    

    MULTI command is begin transaction and EXEC command is commit transaction

    The result will be returned in order your command queue list.

  14. #TIL : Check vcl file syntax before restarting

    Like NginX, Varnish has a syntax checker function that helps us test the syntactic correctness.

    $ varnishd -C -f [vcl file path]

    Varnish will compile the file and output the result to stdout. If something goes wrong, it will throw a message like

    > Message from VCC-compiler:
    > Expected an action, 'if', '{' or '}'
    > ('input' Line 74 Pos 6)
    >     vcl_hash(req.http.Cookie);
    > -----########------------------
    >
    > Running VCC-compiler failed, exit 1
    
  15. #TIL : Using mark to bookmark checkpoints in files

    Bookmarking a checkpoint will help you get back to it intermidately. Ex: your have to find some text to replace something but want to return back current position.

    Set a mark

    • [NORMAL MODE] , type m then follow by a letter from a-z (lowercase is filescope, uppercase for global scope - vim scope)

    Go to a mark

    • [NORMAL MODE] , type backstick ` then follow by the letter your marked above.

    List all current marks

    • [NORMAL MODE], :marks
    • It shows all marks included special ones :
    CommandDescription
    `.jump to position where last change occurred in current buffer
    `"jump to position where last exited current buffer
    `0jump to position in last file edited (when exited Vim)
    ''jump back (to line in current buffer where jumped from)
    ``jump back (to position in current buffer where jumped from)

    TIPS : Can use it as a motion with change, delete or yank

  16. #TIL : Basics of Elasticsearch

    Last days, I developed a EFK stack to centralize my system logging. I really like the concepts of FluentD, it's better than original stack ELK of elastic company.

    So I need to learn basics about Elasticsearch and Kibana

    This is what I learned :

    # Get all documents from elasticsearch node
    GET _search
    {
      "query": {
        "match_all": {}
      }
    }
    
    # Check nodes statistics
    GET /_nodes/stats
    
    # Check health of cluster (I don't know why it is always yello status)
    GET _cluster/health
    
    # Get list of indices (indexes)
    GET /_cat/indices?v
    
    # Delete a index (with its data) with name
    DELETE /[index-name]
  17. #TIL : Use journalctl to check system logs

    Logging and Monitoring are important factor for system admin. Checking the log will help you have a closer look into the issue. One tool could help you will handy features is journalctl.

    Here are simple options :

    • -f : follow the log (tailf)
    • -u [service] : filter to show only [service] logs
    • --since=[date] : Show entries not older than the specified date
    • --until=[date] : Show entries not newer than the specified date

    Example :

    $ sudo journalctl -u nginx.service
    $ sudo journalctl -u nginx.service --since yesterday
    $ sudo journalctl -u nginx.service --since "2018-01-01" --until today
  18. #TIL : Ansible running host pattern

    Ansible supports pattern to select and filter running hosts from all hosts. Here is some common pattern

    • * : wildcard, standalone mean all
    • group1,group2 : run hosts belong to group1 or group2
    • *.web : run hosts belongs to group matches pattern. Ex: backend.web, frontend.web
    • all:!abc : run all hosts exclude hosts belongs to group abc

    Infrastructure by code ;)

  19. #TIL : Set up simple rate limiting on specified port using UFW

    Allow unmetrered connections on networking is so risky. Attacker can use the brute-force attacks to comprosise your service (or simple DOS).

    Linux has a cool firewall to hanlde this, via ip-tables. But it's so complicated to remember all the rule and syntax. That's why UFW was born to save us. :D

    You can use simple command to manage your firewall

    $ ufw default deny incoming # deny any incoming port, should be run before allow any port
    $ ufw default allow outgoing # allow any outgoing port
    $ ufw allow 80 # allow port 80
    $ ufw deny 53/udp # allow udp protocol to port 53
    $ ufw disable # disable firewall
    $ ufw enable # enable firewall
    $ ufw status # check all the rules
    $ ufw delete [num] # delete the rule by its order in status result
    $ ufw reload # reload all rule
    $ ufw limit ssh/tcp # finnaly, limit ssh (port 22 tcp), deny connections if an IP address has attempted to initiate 6 or more connections in the last 30 seconds
  20. #TIL : Disable IPv6 to stop getting stuck in network

    I know IPv6 will be future for networking, but at this moment "It's suck !" :(

    Some service will be failed when trying to connect IPv6 destination :

    • apt package manager
    • smtp
    • curl

    So I decided to disable IPv6 on every production server.

    $ echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
    $ echo "net.ipv6.conf.default.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
    $ echo "net.ipv6.conf.lo.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
    $ 
    $ sudo sysctl -p

    I will re-enable it when everything works perfectly !

  21. #TIL : Tại sao biển xanh lại mặn ? :lol:

    TLDR;

    Biển xanh lại mặn bởi vì đá ở mặt đất cọ xát với mưa trên trời (chứ không phải cá nó đái 😂 )

    Read more : https://oceanservice.noaa.gov/facts/whysalty.html

  22. #TIL : Trigger event after setting val in jQuery

    After setting value of an input via val method, we should call the change chaining method to trigger the onChange event of element.

    $('#selectCity').change(function() {
    	console.log($(this).val());
    });
    
    $('#selectCity').val('HaNoi'); // No trigger
    
    $('#selectCity').val('HoChiMinh').change(); // Fire trigger
  23. #TIL : Tại sao biển xanh lại mặn ? :lol:

    TLDR;

    Biển xanh lại mặn bởi vì đá ở mặt đất cọ xát với mưa trên trời (chứ không phải cá nó đái 😂 )

    Read more : https://oceanservice.noaa.gov/facts/whysalty.html

  24. #TIL : Tracking changes of cookie on webpage

    Using Object.defineProperty helper function as I wrote 3 days ago. We could track the changes of cookie on webpage.

    // Based on Vlad Shevchenko's script at https://stackoverflow.com/a/36826049
    
    var cookieSetterOrig = document.__lookupSetter__("cookie"); // get origin setter function
    var cookieGetterOrig = document.__lookupGetter__("cookie"); // get origin getter function
    Object.defineProperty(document, "cookie", {
        get: function () {
    	console.trace();
            return cookieGetterOrig.apply(document);
        },
        set: function () {
    	console.log(arguments);
    	console.trace();
            return cookieSetterOrig.apply(document, arguments);
        },
        configurable: true
    });

    Notice : This code only works if cookie is changed by javascript, not http header request !

  25. #TIL : Bypass CORS by using JSONP callback

    Sometimes you are blocked from request a cross-origin resource. Instead of adding our domain to allowed list of them, we can use another way to retrieve data from their API by using JSONP (in case they support it).

    The mechanism of JSONP is simple, instead of returning a JSON data. It will return a javascript text with passing your data into a function, whose name is declared in query string. So you just add a new script element with the URL and waiting the callback.

    Example :

    function callMeBaby(data) {
    	console.log(data);
    }
    
    var s = document.createElement("script");
    s.type = "text/javascript";
    s.src = "https://freegeoip.net/json/?callback=callMeBaby";
    document.head.appendChild(s);

    or using jQuery (hide magic)

    $.ajax({
        url: "https://freegeoip.net/json/",
        jsonp: "callback",
        dataType: "jsonp",
        success: function( data ) {
            console.log( data );
        }
    });
  26. #TIL : Define property of an object in hacking way

    Sometimes, we want to define a property of an advanced object (has setter and getter function).

    Now, we could use this helper function Object.defineProperty to define property of an object in a cool way.

    Example :

    const foo = {};
    
    Object.defineProperty(a, 'bar', {
    	value: 'hogehoge',
    	writable: false,
    });
    
    console.log(foo.bar); // 'hogehoge'
    foo.bar = 'foo bar'; // throw an error in strict mode
    console.log(foo.bar); // still be 'hogehoge'

    Modifying setter, getter function

    // Get callstack which function is getting or setting cookie value
    Object.defineProperty(document, 'cookie', {
    	get: function() {
    		console.log('get !');
    		console.trace();
    	},
    	set: function(val) {
    		console.log('set = ', val);
    		console.trace();
    	},
    }); 
  27. #TIL : Debug js code using console.trace

    Browsers provide an useful function help you debug easier than using simple console.log function.

    That is console.trace, which prints a stack trace to called function.

    Example :

    function foo() {
    	var a = 1;
    	bar(a);
    }
    function bar(x) {
    	console.log(x);
    	console.trace();
    }
    
    foo();
  28. #TIL : Sleeping connections in MySQL

    When you check your MySQL process list via command show processlist;, it will show you a useful table which provide all current connection details.

    "Sleep" state connections are most connection pointer waiting for the timeout to terminate. Then they still count as a connection. (Can cause MySQL connection limit error, which default equal 150 connections)

    So next time, remember to close your connection before terminating your app.

    Every connection counts ;)

  29. #TIL : HSTS rule in browser

    HTTP Strict Transport Security (HSTS) is a web security policy mechanism which helps to protect websites against protocol downgrade attacks and cookie hijacking.

    Enabling HSTS on your web will make your browser validate every SSL issues more strictly :

    • User can not visit http version on browser
    • User can not add SSL exception for the domain to ignore the warning. (when SSL cert expire or invalid common name)

    Note : You can manually remove a domain from HSTS in Chrome by accessing this page URL chrome://net-internals/#hsts

    So remember to add HSTS to your website !

  30. #TIL : Create cross-platform downloading app URL

    You have a mobile app for both platforms iOS and Android, each platform has different download URL. But your user doesn't know which platform he using. Clicking wrong URL will lead to user bounce-rate.

    Solution is making only 1 URL to download your app, which can redirect to right place depends on using platform. So how we achieve this ??

    The key of problem is detecting user platform, which can be done by extracting the User-Agent header from http request.

    PlatformUser-Agent pattern
    iOScontains iPhone or iPad or iPod
    Androidcontains Android

    This is how I implement using Caddy web server, you can do same thing in Apache or NGINX

    app.yourcompany.com:443 {
        timeouts 1m
    
        redir 302 {
            if_op or
            if {>User-agent} has iPhone
            if {>User-agent} has iPod
            if {>User-agent} has iPad
    	/ [apple-store-url]
        }
    
        redir 302 {
            if {>User-agent} has Android
    	/ [google-play-url]
        }
    
        redir 302 {
            if {>User-agent} not_has iPhone
            if {>User-agent} not_has iPod
            if {>User-agent} not_has iPad
            if {>User-agent} not_has Android
            / [your-landing-page-which-user-visit-on-desktop-device]
        }
    }
    

    Then we get cool and easy to remember link, right ?

  31. #TIL : Using web proxy to bypass firewalls

    Someday, you will be blocked by a firewall while trying crawling or accessing some website. The reason is they block your IP address from accessing the server.

    One solution is using a web proxy (http proxy, socks4 or socks5) to bypass the firewall, by adding the middle-man server between you and target. It's a bit unsecured but you could use for https site only.

    Some HTTP Proxy supports https will stream TLS data from target to you (so don't worry about proxy server can read you data). Btw, it only knows which domain and IP address you're connecting.

    To find a free proxy from the internet, try this service : https://gimmeproxy.com/

    It provides a cool API to fetch new proxy from its database.

    Example this endpoint will return JSON response including proxy anonymous, supports HTTPS, from Japan and minimum speed more than 100KB

    http://gimmeproxy.com/api/getProxy?anonymityLevel=1&supportsHttps=false&country=JP&minSpeed=100 
    

    In case you need more requests per day, try a subscription (cancelable and refundable). I tried last days, and really like their service (although I cancelled subscription b/c I don't need proxy anymore).

    Break the rules ! ;)