#TIL : HTTP2 supported for python requests library

The sophisticated http client in Python is requests, it has simple API but powerful features. You can use it for crawling, sending request to third-party API or writing tests.

Btw, at this moment it doesn’t support HTTP/2 protocol (actually we often doesn’t need its Server Push or Multi resource stream features). But sometime the API endpoint only supports HTTP/2 like Akamai Load Balacing service.

The hero is new library named hyper, it has been developing to support full HTTP/2 specs. But if all we need is requesting single request to a HTTP/2 server. It works like a charm.

Installation

1
2
$ pip install requests
$ pip install hyper

Usage

1
2
3
4
5
6
7
import requests
from hyper.contrib import HTTP20Adapter
s = requests.Session()
s.mount('https://', HTTP20Adapter())
r = s.get('https://cloudflare.com/')
print(r.status_code)
print(r.url)

This mean any url has prefix https:// will be hanlded by HTTP20Adaper of hyper library

Notice

If you run above example, you will see the result

1
2
200
https://cloudflare.com/

While you expected it would auto-follow redirect to the page https://www.cloudflare.com/

We can fix it by using the newer version than 0.7.0 to fix the header key bytestring issue

1
2
$ pip uninstall hyper
$ pip install https://github.com/Lukasa/hyper/archive/development.zip

Then try it out !!!

Read Input Data From USB Devices in Linux

Introduction

You just bought a new input device like barcode scanner, proximity card reader with USB port. And you want expose only its data to a file or send to server for processing ? But it acts like a keyboard, type the characters then trigger Enter key.

This tutorial can help you make a program can exclusive receive input data from the device and don’t affect to another program.

The program is written in Python and can run on almost Linux system (Ubuntu desktop, Raspberry Pi, Arduino, etc..)

Instruction

Install dependencies

On a Debian compatible OS:

1
2
$ apt-get install python-dev python-pip gcc
$ apt-get install linux-headers-$(uname -r)

On a Redhat compatible OS:

1
2
$ yum install python-devel python-pip gcc
$ yum install kernel-headers-$(uname -r)

On Arch Linux and derivatives:

1
$ pacman -S core/linux-api-headers python-pip gcc

Then, install evdev package via pip

1
$ sudo pip install evdev

Download program sourcecode

Copy this file and save as main.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
#!/usr/bin/python

import sys
from evdev import InputDevice, list_devices, ecodes, categorize

CODE_MAP_CHAR = {
'KEY_MINUS': "-",
'KEY_SPACE': " ",
'KEY_U': "U",
'KEY_W': "W",
'KEY_BACKSLASH': "\\",
'KEY_GRAVE': "`",
'KEY_NUMERIC_STAR': "*",
'KEY_NUMERIC_3': "3",
'KEY_NUMERIC_2': "2",
'KEY_NUMERIC_5': "5",
'KEY_NUMERIC_4': "4",
'KEY_NUMERIC_7': "7",
'KEY_NUMERIC_6': "6",
'KEY_NUMERIC_9': "9",
'KEY_NUMERIC_8': "8",
'KEY_NUMERIC_1': "1",
'KEY_NUMERIC_0': "0",
'KEY_E': "E",
'KEY_D': "D",
'KEY_G': "G",
'KEY_F': "F",
'KEY_A': "A",
'KEY_C': "C",
'KEY_B': "B",
'KEY_M': "M",
'KEY_L': "L",
'KEY_O': "O",
'KEY_N': "N",
'KEY_I': "I",
'KEY_H': "H",
'KEY_K': "K",
'KEY_J': "J",
'KEY_Q': "Q",
'KEY_P': "P",
'KEY_S': "S",
'KEY_X': "X",
'KEY_Z': "Z",
'KEY_KP4': "4",
'KEY_KP5': "5",
'KEY_KP6': "6",
'KEY_KP7': "7",
'KEY_KP0': "0",
'KEY_KP1': "1",
'KEY_KP2': "2",
'KEY_KP3': "3",
'KEY_KP8': "8",
'KEY_KP9': "9",
'KEY_5': "5",
'KEY_4': "4",
'KEY_7': "7",
'KEY_6': "6",
'KEY_1': "1",
'KEY_0': "0",
'KEY_3': "3",
'KEY_2': "2",
'KEY_9': "9",
'KEY_8': "8",
'KEY_LEFTBRACE': "[",
'KEY_RIGHTBRACE': "]",
'KEY_COMMA': ",",
'KEY_EQUAL': "=",
'KEY_SEMICOLON': ";",
'KEY_APOSTROPHE': "'",
'KEY_T': "T",
'KEY_V': "V",
'KEY_R': "R",
'KEY_Y': "Y",
'KEY_TAB': "\t",
'KEY_DOT': ".",
'KEY_SLASH': "/",
}

def parse_key_to_char(val):
return CODE_MAP_CHAR[val] if val in CODE_MAP_CHAR else ""

if __name__ == "__main__":
print "List of your devices :"
devices = [InputDevice(fn) for fn in list_devices()]
for device in devices:
print "\t{}\t{}".format(device.fn, device.name)

print "Choose event ID :",
event_id = raw_input()

print "Exclusive access to device ? [1 or 0] : ",
exclusive_access = raw_input()

device = InputDevice('/dev/input/event{}'.format(event_id))
if int(exclusive_access) == 1:
device.grab()

for event in device.read_loop():
if event.type == ecodes.EV_KEY:
e = categorize(event)
if e.keystate == e.key_up:
sys.stdout.write(parse_key_to_char(e.keycode))
sys.stdout.flush()

Run program with root permission

1
$ sudo python main.py

Then enter your event id of device based on device name in devices list. If you want exclusive access to that device (means no application can’t access and read the input data), press 1 or 0 to accept or deny.

This is my result, I opened the gedit and typed a paragragh but no character appears in Editor, but they’re shown in the terminal.

Read device input on Linux

Customize output data

You can rewrite the code to send data to web server

1
$ sudo pip install requests
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
#!/usr/bin/python

import sys
import requests
from evdev import InputDevice, list_devices, ecodes, categorize

CODE_MAP_CHAR = {
'KEY_MINUS': "-",
'KEY_SPACE': " ",
'KEY_U': "U",
'KEY_W': "W",
'KEY_BACKSLASH': "\\",
'KEY_GRAVE': "`",
'KEY_NUMERIC_STAR': "*",
'KEY_NUMERIC_3': "3",
'KEY_NUMERIC_2': "2",
'KEY_NUMERIC_5': "5",
'KEY_NUMERIC_4': "4",
'KEY_NUMERIC_7': "7",
'KEY_NUMERIC_6': "6",
'KEY_NUMERIC_9': "9",
'KEY_NUMERIC_8': "8",
'KEY_NUMERIC_1': "1",
'KEY_NUMERIC_0': "0",
'KEY_E': "E",
'KEY_D': "D",
'KEY_G': "G",
'KEY_F': "F",
'KEY_A': "A",
'KEY_C': "C",
'KEY_B': "B",
'KEY_M': "M",
'KEY_L': "L",
'KEY_O': "O",
'KEY_N': "N",
'KEY_I': "I",
'KEY_H': "H",
'KEY_K': "K",
'KEY_J': "J",
'KEY_Q': "Q",
'KEY_P': "P",
'KEY_S': "S",
'KEY_X': "X",
'KEY_Z': "Z",
'KEY_KP4': "4",
'KEY_KP5': "5",
'KEY_KP6': "6",
'KEY_KP7': "7",
'KEY_KP0': "0",
'KEY_KP1': "1",
'KEY_KP2': "2",
'KEY_KP3': "3",
'KEY_KP8': "8",
'KEY_KP9': "9",
'KEY_5': "5",
'KEY_4': "4",
'KEY_7': "7",
'KEY_6': "6",
'KEY_1': "1",
'KEY_0': "0",
'KEY_3': "3",
'KEY_2': "2",
'KEY_9': "9",
'KEY_8': "8",
'KEY_LEFTBRACE': "[",
'KEY_RIGHTBRACE': "]",
'KEY_COMMA': ",",
'KEY_EQUAL': "=",
'KEY_SEMICOLON': ";",
'KEY_APOSTROPHE': "'",
'KEY_T': "T",
'KEY_V': "V",
'KEY_R': "R",
'KEY_Y': "Y",
'KEY_TAB': "\t",
'KEY_DOT': ".",
'KEY_SLASH': "/",
}

def parse_key_to_char(val):
return CODE_MAP_CHAR[val] if val in CODE_MAP_CHAR else ""

if __name__ == "__main__":
print "List of your devices :"
devices = [InputDevice(fn) for fn in list_devices()]
for device in devices:
print "\t{}\t{}".format(device.fn, device.name)

print "Choose event ID :",
event_id = raw_input()

print "Exclusive access to device ? [1 or 0] : ",
exclusive_access = raw_input()

device = InputDevice('/dev/input/event{}'.format(event_id))
if int(exclusive_access) == 1:
device.grab()

data = ""
for event in device.read_loop():
if event.type == ecodes.EV_KEY:
e = categorize(event)
if e.keystate == e.key_up:
if e.keycode == "KEY_ENTER":
requests.post("https://somewhere/code/belongs", {
code: data
})
data = ""
else:
data += parse_key_to_char(e.keycode)

You can read more about evdev API here

Ref :

#TIL : Reference assign object variable

When you have a object x and assign y = x, y will be a ref of x (it looks like pointer of C). So changing property of y means changing property of x.

Ex :

1
2
3
4
x = {"a": 1, "b": 2}
y = x
y['a'] = 100
print x['a'] # Result is 100

So if you want clone the value, use copy lib :

1
2
3
4
5
import copy
x = {"a": 1, "b": 2}
y = copy.deepcopy(x)
y['a'] = 100
print x['a'] # Result is 1

#TIL : String Format Unicode params

1
2
unicode_thing = u"Xin chào mọi người"
a = '{}'.format(unicode_thing)

will cause the error UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 6: ordinal not in range(128)

The solution is add u prefix the pattern (it means using unicode pattern) :

1
2
unicode_thing = u"Xin chào mọi người"
a = u'{}'.format(unicode_thing)