iT邦幫忙

2024 iThome 鐵人賽

DAY 29
2
自我挑戰組

每日挑戰:從零開始的 Python 學習之旅系列 第 29

【Day 29】使用requests模組抓取網路資料

  • 分享至 

  • xImage
  •  

Hi 大家好,

今天要開始介紹Python中的requests模組使用方法,那我們開始吧!

 如果要抓取網路資料,Python提供了一個requests模組,其具備了GETPOST..等網路請求方法,透過此方法可以輕鬆的抓取網路資料

使用Poetry管理套件安裝requests模組(避免汙染外部環境)

1. 建立虛擬環境
 1-1. 執行指令poetry init -n-n取消互動式問答,採用預設設定值
 1-2. 產生管理套件設定檔案pyproject.toml,可以查看安裝了那些套件名稱

PS D:\Project\practice> poetry init -n      
PS D:\Project\practice> 

尚未安裝套件的空白內容

# pyproject.toml
[tool.poetry]
name = "ithome"
version = "0.1.0"
description = ""
authors = ["erichsu <xxxxxxx>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

https://ithelp.ithome.com.tw/upload/images/20240829/20165522Isg6Vnkfbp.png

2. 進入虛擬環境
 2-1. 執行指令poetry shell
 2-2. 如果路徑前面多了類似這樣的提示(practice-py3.12),表示已經進入虛擬環境
 2-3. 會自動生成.venv虛擬環境,同時進入虛擬環境

PS D:\Project\practice> poetry shell       
Creating virtualenv practice in D:\Project\practice\.venv
Spawning shell within D:\Project\practice\.venv
PowerShell 7.4.5
(practice-py3.12) PS D:\Project\practice> 

https://ithelp.ithome.com.tw/upload/images/20240829/20165522mdWm9F6QLi.png

3. 安裝requests模組
 3-1. 安裝指令poetry add requests

(practice-py3.12) PS D:\Project\practice> poetry add requests
Using version ^2.32.3 for requests

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 5 installs, 0 updates, 0 removals

  - Installing certifi (2024.7.4)
  - Installing charset-normalizer (3.3.2)
  - Installing idna (3.8)
  - Installing urllib3 (2.2.2)
  - Installing requests (2.32.3)

Writing lock file
(practice-py3.12) PS D:\Project\practice>

查看pyproject.toml

[tool.poetry]
name = "ithome"
version = "0.1.0"
description = ""
authors = ["erichsu <xxxxxxx>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
requests = "^2.32.3"    <==== 新安裝的套件名稱


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

使用poetry show查看是否安裝成功

(practice-py3.12) PS D:\Project\practice> poetry show
certifi            2024.7.4 Python package for providing Mozilla's CA Bundle.
charset-normalizer 3.3.2    The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
idna               3.8      Internationalized Domain Names in Applications (IDNA)
requests           2.32.3   Python HTTP for Humans.
urllib3            2.2.2    HTTP library with thread-safe connection pooling, file post, and more.
(practice-py3.12) PS D:\Project\practice>

https://ithelp.ithome.com.tw/upload/images/20240829/20165522HyulJ0OHBO.png

使用公開API(臺北市公共自行車即時資訊-YouBike2.0)抓取資料作為範例

公開API: YouBike2.0

其中因為raise_for_status()函數並不會回傳任何結果,所以需要使用try-except來判斷網路抓取資料狀態

import requests

def fetch_youbike_data():
    url = "https://tcgbusfs.blob.core.windows.net/dotapp/youbike/v2/youbike_immediate.json"
    
    try:
        response = requests.get(url)
        response.raise_for_status()  # 檢查請求是否成功
        data = response.json()  # 解析 JSON 資料
        return data
    except Exception as err:
        print(f"Exception : {err}")
    return None

def main():
    youbike_data = fetch_youbike_data()
    
    if youbike_data:
        # 因為所有資料是被包成串列,所以使用for..in方式讀取每一資料
        for station in youbike_data:
            print(f"Station: {station['sna']} - Bikes Available: {station['available_rent_bikes']}")
    else:
        print("Failed to fetch data.")

if __name__ == "__main__":
    main()

最後結果

(practice-py3.12) PS D:\Project\practice> python hi.py
Station: YouBike2.0_辛亥路四段77巷口 - Bikes Available: 2
Station: YouBike2.0_景興路42巷口 - Bikes Available: 1
Station: YouBike2.0_木柵路三段48巷 - Bikes Available: 8
Station: YouBike2.0_木柵路二段109巷 - Bikes Available: 0
Station: YouBike2.0_試院公園 - Bikes Available: 1
Station: YouBike2.0_樟樹公園 - Bikes Available: 8
Station: YouBike2.0_萬美萬安街口 - Bikes Available: 6
Station: YouBike2.0_興隆路二段203巷(敦南) - Bikes Available: 1
Station: YouBike2.0_新光秀明路口 - Bikes Available: 5
Station: YouBike2.0_光輝路134巷口 - Bikes Available: 3
Station: YouBike2.0_光輝公園 - Bikes Available: 4
Station: YouBike2.0_木新公園 - Bikes Available: 2
Station: YouBike2.0_興隆路二段275巷1弄口 - Bikes Available: 6
Station: YouBike2.0_中國科技大學 - Bikes Available: 7
Station: YouBike2.0_樟新街8巷 - Bikes Available: 8
Station: YouBike2.0_仙岩路22巷口 - Bikes Available: 1
Station: YouBike2.0_木柵萬芳路口 - Bikes Available: 5
Station: YouBike2.0_福興路78巷 - Bikes Available: 4
Station: YouBike2.0_辛亥木柵路口 - Bikes Available: 6
Station: YouBike2.0_興泰公園 - Bikes Available: 3
Station: YouBike2.0_興岩社會福利綜合大樓 - Bikes Available: 18
Station: YouBike2.0_興隆路二段22巷9弄口 - Bikes Available: 0
Station: YouBike2.0_木新路三段155巷口 - Bikes Available: 8
Station: YouBike2.0_景興路102巷口 - Bikes Available: 2
Station: YouBike2.0_景福街223巷 - Bikes Available: 0
Station: YouBike2.0_達賢圖書館 - Bikes Available: 9
Station: YouBike2.0_捷運木柵站(西側) - Bikes Available: 10
Station: YouBike2.0_臺北自來水事業處 - Bikes Available: 1
Station: YouBike2.0_汀州思源街口東南側 - Bikes Available: 2
Station: YouBike2.0_捷運公館站(1號出口) - Bikes Available: 2
Station: YouBike2.0_捷運公館站(4號出口) - Bikes Available: 11
Station: YouBike2.0_捷運古亭站(9號出口) - Bikes Available: 5
Station: YouBike2.0_捷運古亭站(2號出口) - Bikes Available: 13
......
......

那今天就介紹到這裡,我們明天見~


上一篇
【Day 28】進階檔案處理篇 - 續集
下一篇
【Day 30】常用內建模組篇
系列文
每日挑戰:從零開始的 Python 學習之旅30
圖片
  直播研討會
圖片
{{ item.channelVendor }} {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

尚未有邦友留言

立即登入留言