2024 iThome 鐵人賽

DAY 30

Python

Python和R入門語法比較系列第 30 篇

15 [R] for 迴圈和 html網頁資料解析 httr, xml2, [16th 鐵人 Day 30]

16th鐵人賽

carplee

團隊為你抓鯉魚

2024-10-13 11:40:25

78 瀏覽

分享至

下載response.rds

複習for迴圈

way 1

for (i in 1:5){

}

for (i in 1:5){
  print(i)
}

[1] 1
[1] 2
[1] 3
[1] 4
[1] 5

way 2

for (i in 向量){

}

c = c(1:5)

for (i in c){
  print(i)
}

[1] 1
[1] 2
[1] 3
[1] 4
[1] 5

用R解析網頁資料網頁資料抓取回來會是response格式

讀進response格式的檔案

res = readRDS("data/response.rds")

res

Response [https://www.ptt.cc/bbs/Baseball/index17256.html]
  Date: 2024-09-23 12:29
  Status: 200
  Content-Type: text/html; charset=utf-8
  Size: 22 kB
<!DOCTYPE html>
<html>
	<head>
		<meta charset="utf-8">
		

<meta name="viewport" content="w...

<title>看板 Baseball 文章列表 - 批踢踢實業坊...

...

class(res)

下載 httr 套件解析response格式成為xml_document格式

install.packages('httr')

library(httr)
xml_doc = content(res)
xml_doc

{html_document}
<html>
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=U ...
[2] <body>\n\t\t\n<div id="topbar-container">\n\t<div id="topbar" class=" ...

class(xml_doc)

[1] "xml_document" "xml_node"

下載 xml2套件解析xml_document格式

install.packages('xml2')

library(xml2)

當資料已成為xml_document格式

選取 html 的 <head>標籤

xml_find_all(xml_doc, 'head')

{xml_nodeset (1)}
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=U ...

選取 html 的 <body>標籤

xml_find_all(xml_doc, 'body')

{xml_nodeset (1)}
[1] <body>\n\t\t\n<div id="topbar-container">\n\t<div id="topbar" class=" ...

選取 html 的 <div>標籤

xml_find_all(xml_doc, '//div')

{xml_nodeset (249)}
 [1] <div id="topbar-container">\n\t<div id="topbar" class="bbs-content"> ...
 [2] <div id="topbar" class="bbs-content">\n\t\t<a id="logo" href="/bbs/" ...
 [3] <div id="main-container">\n\t<div id="action-bar-container">\n\t\t<d ...
 [4] <div id="action-bar-container">\n\t\t<div class="action-bar">\n\t\t\ ...
 [5] <div class="action-bar">\n\t\t\t<div class="btn-group btn-group-dir" ...
 [6] <div class="btn-group btn-group-dir">\n\t\t\t\t<a class="btn selecte ...
 [7] <div class="btn-group btn-group-paging">\n\t\t\t\t<a class="btn wide ...
 [8] <div class="r-list-container action-bar-margin bbs-screen">\n\t\t<di ...
 [9] <div class="search-bar">\n\t\t\t<form type="get" action="search" id= ...
[10] <div class="r-ent">\n\t\t\t<div class="nrec"><span class="hl f3">11< ...
[11] <div class="nrec"><span class="hl f3">11</span></div>
[12] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[13] <div class="meta">\n\t\t\t\t<div class="author">siugwai</div>\n\t\t\ ...
[14] <div class="author">siugwai</div>
[15] <div class="article-menu">\n\t\t\t\t\t\n\t\t\t\t\t<div class="trigge ...
[16] <div class="trigger">⋯</div>
[17] <div class="dropdown">\n\t\t\t\t\t\t<div class="item"><a href="/bbs/ ...
[18] <div class="item"><a href="/bbs/Baseball/search?q=thread%3A%5B%E6%96 ...
[19] <div class="item"><a href="/bbs/Baseball/search?q=author%3Asiugwai"> ...
[20] <div class="date"> 9/23</div>

增加選取的條件

xml_find_all(xml_doc, '//div[@class="title"]')

{xml_nodeset (20)}
 [1] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [2] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [3] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [4] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [5] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [6] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [7] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [8] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
 [9] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[10] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[11] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[12] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[13] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[14] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[15] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[16] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[17] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[18] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172708 ...
[19] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172709 ...
[20] <div class="title">\n\t\t\t\n\t\t\t\t<a href="/bbs/Baseball/M.172709 ...

xml_find_all(xml_doc, '//div[@class="title"]/a')

{xml_nodeset (20)}
 [1] <a href="/bbs/Baseball/M.1727085626.A.28F.html">[新聞] 大谷翔平前口譯水原辯護團隊 要 ...
 [2] <a href="/bbs/Baseball/M.1727085798.A.C06.html">[新聞] 12強賽／賽事口號「就一起」  ...
 [3] <a href="/bbs/Baseball/M.1727085833.A.9C4.html">[分享] 台南Josh:辜仲諒在12強記 ...
 [4] <a href="/bbs/Baseball/M.1727086105.A.FA0.html">[新聞] 江少慶傷勢看三間醫院評估中　本 ...
 [5] <a href="/bbs/Baseball/M.1727086125.A.2FC.html">[新聞] 中職》引退賽後隔天 周思齊降二 ...
 [6] <a href="/bbs/Baseball/M.1727086236.A.13B.html">[新聞] 潘威倫928先發出賽局數拿捏是 ...
 [7] <a href="/bbs/Baseball/M.1727086451.A.4BF.html">[新聞] 辜仲諒批職棒拖累台灣棒球　蔡其 ...
 [8] <a href="/bbs/Baseball/M.1727086680.A.7CA.html">[情報] 今日村上宗隆 逆轉2分砲</a>
 [9] <a href="/bbs/Baseball/M.1727086702.A.333.html">Re: [討論] 臺灣可能的第二顆巨蛋新 ...
[10] <a href="/bbs/Baseball/M.1727086824.A.34F.html">[新聞] 演唱會壓縮大巨蛋棒球比賽場次？ ...
[11] <a href="/bbs/Baseball/M.1727086921.A.EC9.html">[討論] 單論職棒聯盟，中職世界排第幾？ ...
[12] <a href="/bbs/Baseball/M.1727086953.A.3A2.html">[新聞] 讓林安可拚外野金手套、最佳十人 ...
[13] <a href="/bbs/Baseball/M.1727086998.A.56D.html">[新聞] SPC 剛為周思齊引退獻聲 田 ...
[14] <a href="/bbs/Baseball/M.1727087106.A.D25.html">[討論] 邀請林威助當教練可以讓中職更進 ...
[15] <a href="/bbs/Baseball/M.1727087128.A.330.html">[討論] 大家對嘟嘟引退賽出席教練們有哪 ...
[16] <a href="/bbs/Baseball/M.1727087290.A.D81.html">[討論] 周思齊退休當辜辜特助是不是可惜 ...
[17] <a href="/bbs/Baseball/M.1727088686.A.B30.html">[新聞] 獅隊好消息！潘傑楷檢查是舊傷發 ...
[18] <a href="/bbs/Baseball/M.1727089545.A.0FB.html">[情報] 今日イチロー</a>
[19] <a href="/bbs/Baseball/M.1727090895.A.D45.html">Re: [新聞] 辜仲諒批職棒拖累台灣棒 ...
[20] <a href="/bbs/Baseball/M.1727091437.A.B41.html">[分享] 今日松井秀喜</a>

a = xml_find_all(xml_doc, '//div[@class="title"]/a')

用xml_attr() 取得 <a>標籤的屬性(attribute):href

xml_attr(a, 'href')

 [1] "/bbs/Baseball/M.1727085626.A.28F.html"
 [2] "/bbs/Baseball/M.1727085798.A.C06.html"
 [3] "/bbs/Baseball/M.1727085833.A.9C4.html"
 [4] "/bbs/Baseball/M.1727086105.A.FA0.html"
 [5] "/bbs/Baseball/M.1727086125.A.2FC.html"
 [6] "/bbs/Baseball/M.1727086236.A.13B.html"
 [7] "/bbs/Baseball/M.1727086451.A.4BF.html"
 [8] "/bbs/Baseball/M.1727086680.A.7CA.html"
 [9] "/bbs/Baseball/M.1727086702.A.333.html"
[10] "/bbs/Baseball/M.1727086824.A.34F.html"
[11] "/bbs/Baseball/M.1727086921.A.EC9.html"
[12] "/bbs/Baseball/M.1727086953.A.3A2.html"
[13] "/bbs/Baseball/M.1727086998.A.56D.html"
[14] "/bbs/Baseball/M.1727087106.A.D25.html"
[15] "/bbs/Baseball/M.1727087128.A.330.html"
[16] "/bbs/Baseball/M.1727087290.A.D81.html"
[17] "/bbs/Baseball/M.1727088686.A.B30.html"
[18] "/bbs/Baseball/M.1727089545.A.0FB.html"
[19] "/bbs/Baseball/M.1727090895.A.D45.html"
[20] "/bbs/Baseball/M.1727091437.A.B41.html"

用xml_text() 取得內文

xml_text(a)

 [1] "[新聞] 大谷翔平前口譯水原辯護團隊 要求量刑宣"     
 [2] "[新聞] 12強賽／賽事口號「就一起」 中職6隊12女"    
 [3] "[分享] 台南Josh:辜仲諒在12強記者會發言逐字"       
 [4] "[新聞] 江少慶傷勢看三間醫院評估中　本季不出"      
 [5] "[新聞] 中職》引退賽後隔天 周思齊降二軍"           
 [6] "[新聞] 潘威倫928先發出賽局數拿捏是挑戰 餅總"      
 [7] "[新聞] 辜仲諒批職棒拖累台灣棒球　蔡其昌反駁："    
 [8] "[情報] 今日村上宗隆 逆轉2分砲"                    
 [9] "Re: [討論] 臺灣可能的第二顆巨蛋新聞整理"          
[10] "[新聞] 演唱會壓縮大巨蛋棒球比賽場次？ 北市體"     
[11] "[討論] 單論職棒聯盟，中職世界排第幾？"            
[12] "[新聞] 讓林安可拚外野金手套、最佳十人 餅總："     
[13] "[新聞] SPC 剛為周思齊引退獻聲 田中將大本季首"     
[14] "[討論] 邀請林威助當教練可以讓中職更進步嗎?"       
[15] "[討論] 大家對嘟嘟引退賽出席教練們有哪些印象"      
[16] "[討論] 周思齊退休當辜辜特助是不是可惜了？"        
[17] "[新聞] 獅隊好消息！潘傑楷檢查是舊傷發炎已開始"    
[18] "[情報] 今日イチロー"                              
[19] "Re: [新聞] 辜仲諒批職棒拖累台灣棒球　蔡其昌反駁："
[20] "[分享] 今日松井秀喜"

15 [Python] for 迴圈和 html網頁資料解析 by bs4套件(BeautifulSoup) [16th 鐵人 Day 29]

系列文

Python和R入門語法比較共 30 篇

RSS系列文訂閱系列文

1 人訂閱

完整目錄

直播研討會

{{ item.channelVendor }} {{ item.webinarstarted }} |

直播中

尚未有邦友留言

立即登入留言

參賽組數

1064 組

團體組數

40 組

累計文章數

21841 篇

完賽人數

595 人

15th鐵人賽 16th鐵人賽 13th鐵人賽 14th鐵人賽 12th鐵人賽 11th鐵人賽鐵人賽 2019鐵人賽 javascript 2018鐵人賽 python 2017鐵人賽 windows php c# windows server linux css react vue.js

IT邦幫忙

Python和R入門語法比較系列 第 30 篇

15 [R] for 迴圈 和 html網頁資料解析 httr, xml2, [16th 鐵人 Day 30]