Changing the selenium user agent in python
Posted on: January 13, 2024
User agent is a one of the element in request headers and it includes browser information, operating system and device. Selenium is most popular tool for testing and web scraping. While sending a request with selenium sites easily block due to user agent element will send a bot. To avoid blocking we change the user agent manually.
Import all the necessary libraries
from selenium import webdriver
User agent example
user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
Final code:
def page_source_with_selenium(link):
options = webdriver.ChromeOptions()
options.add_argument(f"--user-agent={user_agent}")
driver = webdriver.Chrome(options=options)
driver.get(link)
Find a right dataset that you are looking for from crawl feeds store.
Submit data request if not able to find right dataset.
Custom request