資料的壓縮最好是能做在前端,因為網路最慢的地方就是前後端的溝通了,現在的壓縮演算法有很多,舉凡gzip、PKzip、LZW、zlib等等
我們的Lab追求簡單,於是使用了python本身就有的zlib,壓縮和解壓縮的使用非常簡單,如下
def compress(data):
return zlib.compress(data)
def decompress(data):
print(data)
return zlib.decompress(data)
https://github.com/kaichiachen/python-simple-object-storage/blob/master/utils/utils.py#L53
然後我們分別在upload object的時候進行壓縮
content = compress(content.encode()) # content type will be byte after compress
hash = sha256(content)
size = len(content)
chunk = 10
for i in range(0, size, chunk):
res = requests.post(f"{server_url}/objects/{name}", files={
'obj': content[i:i+chunk],
'hash':hash,
'version': args.version,
'start': i,
'length': chunk,
'size': size})
以及download的時候解壓縮
while len(content)<size:
try:
content += requests.get(f"{server_url}/objects/{name}?version={version}&start={start}").content
except KeyboardInterrupt:
sys.exit()
except Exception as e:
start = len(content)
print('Something wrong, downloading again...', e)
try:
print(f'Get content: {decompress(content).decode()}')
except Exception as e:
print(f'Something wrong: {e}')
於是乎,從使用者的角度來看,看起來就只是上傳下載Object而已,性能的優化都在前端做好了