Home < Changing the selenium user agent in python

Changing the selenium user agent in python

Posted on: January 13, 2024

User agent is a one of the element in request headers and it includes browser information, operating system and device. Selenium is most popular tool for testing and web scraping. While sending a request with selenium sites easily block due to user agent element will send a bot. To avoid blocking we change the user agent manually.

Import all the necessary libraries

from selenium import webdriver

User agent example

user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"

Final code:

def page_source_with_selenium(link):
    options = webdriver.ChromeOptions()
    options.add_argument(f"--user-agent={user_agent}")
    driver = webdriver.Chrome(options=options)
    driver.get(link)

 

Find a right dataset that you are looking for from crawl feeds store.

Datasets

Submit data request if not able to find right dataset.
Custom request