iT邦幫忙

2021 iThome 鐵人賽

DAY 25
0
永豐金融APIs

深入解析 Shioaji API系列 第 25

Day 25 - 重覆呼叫shioaji.Shioaji()產生的記憶體問題-修正篇

在前一天的Day 24 - Shiaoji.Login踩坑經驗及修正中,談到在執行login動作時,未考慮要等待contract fetch動作完成,導致所抓的資料內容不完整。這次的踩坑經驗,也讓我重新去思考Day 23 - 重覆呼叫shioaji.Shioaji()產生的記憶體問題中,是否也會因為相同的問題,影響測試的結果。
所以,我把當時測試的程式碼,改為以下方式:

import os, psutil, gc, time
import shioaji as sj
from dotenv import load_dotenv

load_dotenv('D:\\python\\shioaji\\.env') #讀取.env中的環境變數

process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
for i in range(5):
    api = sj.Shioaji(simulation=True)
    print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
    start_time = time.time()
    # api.login(
    #     person_id=os.getenv('YOUR_PERSON_ID'), 
    #     passwd=os.getenv('YOUR_PASSWORD'),
    #     contracts_timeout=10000
    # )
    api.login(
        person_id='PAPIUSER01',
        passwd='2222'
    )
    print(api.Contracts) #增加此行,等待所有的Contract資料fetch完成後,才繼續
    print(f'contract exe time:{time.time()-start_time}')
    print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
    api.logout()
    del api

gc.collect()
print(f'mem usage after del api: {process.memory_info()[0]/float(2 ** 20)}MB')

這次的程式中,在login後增加一行print(api.Contracts),確保所有的Contract資料都fetch完成後,才繼續後續的動作;另外在記憶體計算部份,因為換算時是除以2的20次方,所以單位是MB而不是KB。而這一次的測試,也一併測試Contract fetch所需要的時間,以便與優化後做比較。

修正後的測試結果

本次的測試,也依照上次的4個不同場景做測試,測試結果如下:

程式點 windows&正式環境 windows&測試環境 ubuntu&正式環境 ubuntu&測試環境
start 33.984375 33.9296875 34.55859375 34.4375
0-initial 37.87890625 38.02734375 38.0234375 37.8046875
0-login 226.46875 231.0390625 220.9765625 39.03515625
0-contract fetch time 7.203447580337524 8.030523300170898 5.9245991706848145 7.924567461013794
1-initial 226.4453125 231.01171875 220.9765625 229.625
1-login 406.69921875 409.90625 219.4492188 400.9609375
1-contract fetch time 7.261681318283081 8.347200632095337 6.103384017944336 8.131371021270752
2-initial 406.640625 413.8671875 398.80078125 415.3203125
2-login 587.5625 597.703125 586.6640625 597.3125
2-contract fetch time 6.988409757614136 8.512909173965454 6.160722017288208 7.1584250926971436
3-initial 587.51171875 597.69140625 586.6640625 597.3125
3-login 767.63671875 780.59765625 774.5546875 781.94921875
3-contract fetch time 7.1759161949157715 8.046003103256226 6.2177183628082275 7.363017797470093
4-initial 767.58203125 780.578125 774.5546875 781.94921875
4-login 950.8515625 963.15234375 954.59375 970.33203125
4-contract fetch time 7.381143569946289 8.393444299697876 6.169715166091919 7.520856142044067
del & gc 950.81640625 963.05078125 954.59375 970.33203125

這一次的測試結果,每個測試場景的記憶體使用量及時間,其實都差不多;而上一次的測試結果,測試環境及Linux環境下,之所以都會比windows平台或是正式環境要來得低,都是因為Contract資料下載不完整,而得到錯誤的測試結果
雖然在測試程式中,有執行del api及gc.collect,但測試的結果卻是記憶體使用量沒有下降,因為Python的記憶體空間都是由Python memory manager在做管理,而實際上會不會清理出記憶體空間,也是要看是不是符合memory manager的管理機制,並不像C/C++可以直接在程式中做記憶體管理。
關於Python memory manager,可參考下列文章:
Python 3.10.0 Documentation » Python/C API Reference Manual » Memory Management
Memory Management in Python
Python 用del删除变量以后为什么还是OOM(Python的内存管理与垃圾回收)

優化方式及差異

在上次那篇有談到,記憶體使用量優化的方式,這裡就說明程式碼調整內容,以及優化後的測試結果
優化的分別程式碼如下:

將sj.Shioaji()移至外層,不重覆執行

import os, psutil, time
import shioaji as sj
from dotenv import load_dotenv

load_dotenv('D:\\python\\shioaji\\.env')

process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
process_start_time = time.time()
api = sj.Shioaji() #api宣告移至最外層,不重覆執行
for i in range(5):
    print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
    start_time = time.time()
    api.login(
        person_id=os.getenv('YOUR_PERSON_ID'), 
        passwd=os.getenv('YOUR_PASSWORD')
    )
    print(api.Contracts)
    print(f'({i})contract fetch time:{time.time()-start_time}')
    print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
    api.logout()

print(f'process total exe time:{time.time()-process_start_time}')

僅第一次執行時,執行contract fetch並將Contracts資料儲存

import os, psutil, time
import shioaji as sj
from dotenv import load_dotenv

load_dotenv('D:\\python\\shioaji\\.env')

process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
process_start_time = time.time()
my_contract = None #宣告my_contract物件

for i in range(5):
    api = sj.Shioaji()
    print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
    start_time = time.time()
    # 當my_contract為None時,執行contract fetch動作
    if my_contract is None:
        api.login(
            person_id=os.getenv('YOUR_PERSON_ID'), 
            passwd=os.getenv('YOUR_PASSWORD')
        )
        print(api.Contracts) #等待Contract fetch完成
        my_contract = api.Contracts #將my_contract指向api.Contracts
    else:
        print('fetch_contract=False...')
        api.login(
            person_id=os.getenv('YOUR_PERSON_ID'), 
            passwd=os.getenv('YOUR_PASSWORD'),
            fetch_contract=False #設為False,表示login後不執行contract fetch
        )
        print(my_contract)
    print(f'({i})contract fetch time:{time.time()-start_time}')
    print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
    api.logout()

print(f'process total exe time:{time.time()-process_start_time}')

將sj.Shioaji()移至外層,不重覆執行,以及僅第一次執行fetch動作

import os, psutil, time
import shioaji as sj
from dotenv import load_dotenv

load_dotenv('D:\\python\\shioaji\\.env') #讀取.env中的環境變數

process = psutil.Process(os.getpid())
print(os.getppid())
print(f'mem usage as start: {process.memory_info()[0]/float(2 ** 20)}MB')
process_start_time = time.time()
api = sj.Shioaji() #api宣告移至最外層,不重覆執行

for i in range(5):
    print(f'({i})mem usage at api initial: {process.memory_info()[0]/float(2 ** 20)}MB')
    start_time = time.time()
    # 當api.Contracts為None時,執行contract fetch動作
    if api.Contracts is None:
        api.login(
            person_id=os.getenv('YOUR_PERSON_ID'), 
            passwd=os.getenv('YOUR_PASSWORD')
        )
    else:
        print('fetch_contract=False...')
        api.login(
            person_id=os.getenv('YOUR_PERSON_ID'), 
            passwd=os.getenv('YOUR_PASSWORD'),
            fetch_contract=False #設為False,表示login後不執行contract fetch
        )
    print(api.Contracts)
    print(f'({i})contract fetch time:{time.time()-start_time}')
    print(f'({i})mem usage at api.login: {process.memory_info()[0]/float(2 ** 20)}MB')
    api.logout()

print(f'process total exe time:{time.time()-process_start_time}')

測試結果

程式點 sj.Shioaji()移至外層 僅第一次執行時,執行contract fetch並將Contracts資料儲存 sj.Shioaji()移至外層+僅第一次fetch
start 34.0390625 33.9765625 33.9765625
0-initial 37.97265625 37.84765625 38.01171875
0-login 226.421875 226.765625 226.22265625
0-contract fetch time 6.864650726318359 7.08452582359314 6.8395609855651855
1-initial 226.4453125 226.91015625 226.16015625
1-login 247.984375 227.140625 226.31640625
1-contract fetch time 1.8755991458892822 1.5779919624328613 1.5284152030944824
2-initial 249.35546875 227.18359375 226.265625
2-login 250.640625 227.34765625 226.4375
2-contract fetch time 2.2606329917907715 1.7178151607513428 1.3352644443511963
3-initial 259.97265625 227.33203125 226.39453125
3-login 273.60546875 227.5 226.5625
3-contract fetch time 2.4294867515563965 1.4267780780792236 1.6401028633117676
4-initial 274.13671875 227.48828125 226.5078125
4-login 276.0234375 227.6484375 226.671875
4-contract fetch time 2.1341898441314697 1.4912939071655273 1.2037618160247803
process total exe time 16.376935720443726 13.961941003799438 12.832036018371582

可以看到,這三種優化方式,從第二次登入後的測試結果,記憶體使用量或是contract fetch time,都跟優化前的有很明顯示的改善。


上一篇
Day 24 - Shiaoji.Login踩坑經驗及修正
下一篇
Day 26 - 建立自己的K線資料庫 (上)
系列文
深入解析 Shioaji API30

尚未有邦友留言

立即登入留言