PHPFixing
  • Privacy Policy
  • TOS
  • Ask Question
  • Contact Us
  • Home
  • PHP
  • Programming
  • SQL Injection
  • Web3.0

Saturday, October 15, 2022

[FIXED] How do you use BeautifulSoup to fetch data in specific format?

 October 15, 2022     beautifulsoup, dom, web-scraping     No comments   

Issue

I would create a python code to fetch the average volume of a given link stock using BeautifulSoup.

What I have done so far:

import bs4
import requests
from bs4 import BeautifulSoup

r=requests.get('https://finance.yahoo.com/quote/M/key-statistics?p=M')
soup=BeautifulSoup(r.content,"html.parser")

# p = soup.find_all(class_="Fw(500) Ta(end) Pstart(10px) Miw(60px)")[1].get_text
# p = soup.find_all('td')[2].get_text     
# p = soup.find_all('table', class_='W(100%) Bdcl(c)')[70].tr.get_text

Anyway, I was able to get that number directly from google console using this command:

Document.querySelectorAll('table tbody tr td')[71].innerText
"21.07M"

Please, help with the basic explanation, I know a few about DOM.


Solution

You can use this logic to make it easier:

  • Use find_all to find all the spans in the html file
  • Search the spans for the correct label (Avg Vol...)
  • Use parent to go up the hierarchy to the full table row
  • Use find_all again from the parent to get the last cell which contains the value

Here is the updated code:

import bs4
import requests
from bs4 import BeautifulSoup

r=requests.get('https://finance.yahoo.com/quote/M/key-statistics?p=M')
soup=BeautifulSoup(r.content,"html.parser")

p = soup.find_all('span')
for s in p:   # each span
   if s.text == 'Avg Vol (10 day)':  # starting cell
       pnt = s.parent.parent  # up 2 levels, table row
       print(pnt.find_all('td')[-1].text)  # last table cell

Output

21.76M


Answered By - Mike67
Answer Checked By - Dawn Plyler (PHPFixing Volunteer)
  • Share This:  
  •  Facebook
  •  Twitter
  •  Stumble
  •  Digg
Newer Post Older Post Home

0 Comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Total Pageviews

Featured Post

Why Learn PHP Programming

Why Learn PHP Programming A widely-used open source scripting language PHP is one of the most popular programming languages in the world. It...

Subscribe To

Posts
Atom
Posts
Comments
Atom
Comments

Copyright © PHPFixing