I am trying to get the current stock price using fidelity's screener. For example, the current price of AAPL is $165.02 on https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=AAPL.
When I inspect the webpace, the price is shown here: <div _ngcontent-cxa-c16="" class="nre-quick-quote-price">$165.02</div>.
I used this code:
import requestsfrom bs4 import BeautifulSoupdef stock_price(symbol: str = "AAPL") -> str:url = f"https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol={symbol}"response = requests.get(url)soup = BeautifulSoup(response.text, "html.parser")price_tag = soup.find('div', class_='nre-quick-quote-price')current_price = price_tag['value']return current_priceBut got this error:
Traceback (most recent call last):File "get_price.py", line 160, in <module>print(f"Current {symbol:<4} stock price is {stock_price(symbol):>8}")File "get_price.py", line 63, in stock_pricecurrent_price = price_tag['value']TypeError: 'NoneType' object is not subscriptableI also used this code:
from selenium import webdriverdef stock_price(symbol: str = "AAPL") -> str:driver = webdriver.Chrome()url = "https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=" + symboldriver.get(url)current_price = driver.find_element('div.nre-quick-quote-price').textreturn current_priceBut got this error:
Traceback (most recent call last):File "get_price.py", line 103, in <module>price_tag = driver.find_element('div.nre-quick-quote-price')File "C:\Users\X\miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 831, in find_elementreturn self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]File "C:\Users\X\miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 440, in executeself.error_handler.check_response(response)File "C:\Users\X\miniconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 245, in check_responseraise exception_class(message, screen, stacktrace)selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locatorPlease help!
Best Answer
Root cause of your issue: Syntax is not correct. See below.
In your second code, change the below line from:
current_price = driver.find_element('div.nre-quick-quote-price').textTo:
current_price = driver.find_element(By.CSS_SELECTOR, 'div.nre-quick-quote-price').textFull working code:Note: I have added explicitwaits in your code to ensure code is more consistent, this way even if the website is little slow in responding/loading your code will handle it.
from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.support.wait import WebDriverWaitdef stock_price(symbol: str = "AAPL") -> str:driver = webdriver.Chrome()url = "https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=" + symboldriver.get(url)wait = WebDriverWait(driver, 10)current_price = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.nre-quick-quote-price'))).textreturn current_priceprint(stock_price())Result:
$165.02Process finished with exit code 0UPDATED code to run in headless mode:
from selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.support.wait import WebDriverWaitdef stock_price(symbol: str = "AAPL") -> str:options = webdriver.ChromeOptions()options.add_argument('--headless')options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36")driver = webdriver.Chrome(options=options)url = "https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=" + symboldriver.get(url)wait = WebDriverWait(driver, 10)current_price = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.nre-quick-quote-price'))).textreturn current_priceprint(stock_price())