Getting Started with Selenium in Python: A Comprehensive Guide

Selenium lets you control a web browser programmatically — click buttons, fill forms, navigate pages, and extract content. It's widely used for end-to-end testing and web automation. This guide covers Selenium 4 syntax, which differs significantly from earlier versions.

Installation

pip install selenium

Selenium 4.6+ includes Selenium Manager, which automatically downloads the correct browser driver for you. No manual ChromeDriver/GeckoDriver setup required.

For older Selenium versions or finer driver control, use webdriver-manager:

pip install webdriver-manager

Setting Up a WebDriver

Chrome (recommended)

from selenium import webdriver

# Selenium 4.6+ — driver downloaded automatically
driver = webdriver.Chrome()

# With webdriver-manager (Selenium < 4.6)
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

Firefox

driver = webdriver.Firefox()

Headless Mode (no browser window)

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)

Navigating Pages

driver.get("https://example.com")

print(driver.title)        # page title
print(driver.current_url)  # current URL

driver.back()
driver.forward()
driver.refresh()

driver.quit()  # always call this when done

Locating Elements

Selenium 4 uses the By class for all element lookups. The old find_element_by_* methods were removed in Selenium 4.3.

from selenium.webdriver.common.by import By

# Single element
driver.find_element(By.ID, "username")
driver.find_element(By.NAME, "email")
driver.find_element(By.CLASS_NAME, "nav-link")
driver.find_element(By.TAG_NAME, "h1")
driver.find_element(By.LINK_TEXT, "Sign Up")
driver.find_element(By.PARTIAL_LINK_TEXT, "Sign")
driver.find_element(By.CSS_SELECTOR, "input[type='email']")
driver.find_element(By.XPATH, "//button[@type='submit']")

# Multiple elements (returns a list)
links = driver.find_elements(By.TAG_NAME, "a")

Interacting with Elements

# Click
driver.find_element(By.ID, "submit-btn").click()

# Type text
driver.find_element(By.NAME, "username").send_keys("myuser")

# Clear an input first, then type
field = driver.find_element(By.ID, "search")
field.clear()
field.send_keys("selenium python")

# Submit a form
field.submit()

Waiting for Elements

Always use explicit waits instead of time.sleep(). They make your automation more reliable.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

wait = WebDriverWait(driver, timeout=10)

try:
    element = wait.until(
        EC.presence_of_element_located((By.ID, "results"))
    )
    print(element.text)
except TimeoutException:
    print("Element did not appear within 10 seconds")

Common expected conditions:

• presence_of_element_located — element exists in the DOM
• visibility_of_element_located — element is visible
• element_to_be_clickable — element is visible and enabled
• text_to_be_present_in_element — element contains specific text

Selecting Dropdowns

from selenium.webdriver.support.ui import Select

dropdown = Select(driver.find_element(By.ID, "country"))

dropdown.select_by_visible_text("Canada")
dropdown.select_by_value("CA")
dropdown.select_by_index(2)

print(dropdown.first_selected_option.text)

Handling iFrames

# Switch into an iframe
driver.switch_to.frame(driver.find_element(By.ID, "payment_iframe"))

# Now interact with elements inside the iframe
driver.find_element(By.NAME, "card_number").send_keys("4111111111111111")

# Switch back to the main page
driver.switch_to.default_content()

Tabs and Windows

# Open a new tab
driver.execute_script("window.open('');")

# Get all window handles
handles = driver.window_handles

# Switch to the new tab
driver.switch_to.window(handles[1])
driver.get("https://example.com")

# Switch back to the original tab
driver.switch_to.window(handles[0])

Cookies

# Get all cookies
cookies = driver.get_cookies()

# Add a cookie
driver.add_cookie({"name": "token", "value": "abc123"})

# Delete a specific cookie
driver.delete_cookie("session_id")

# Delete all cookies
driver.delete_all_cookies()

Screenshots and JavaScript

# Save a screenshot
driver.save_screenshot("screenshot.png")

# Execute JavaScript
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
driver.execute_script("arguments[0].click();", element)

Practical Example: Search and Scrape

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

try:
    driver.get("https://en.wikipedia.org/wiki/Main_Page")

    search_box = wait.until(EC.element_to_be_clickable((By.ID, "searchInput")))
    search_box.send_keys("Python programming language")
    search_box.send_keys(Keys.ENTER)

    # Wait for the article heading
    heading = wait.until(EC.presence_of_element_located((By.ID, "firstHeading")))
    print(f"Article title: {heading.text}")

    # Get the first paragraph
    intro = driver.find_element(By.CSS_SELECTOR, "#mw-content-text p")
    print(f"Intro: {intro.text[:200]}")
finally:
    driver.quit()

Mobile Emulation

options = webdriver.ChromeOptions()
mobile_emulation = {
    "deviceMetrics": {"width": 390, "height": 844, "pixelRatio": 3.0},
    "userAgent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X)"
}
options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Chrome(options=options)

Conclusion

Selenium 4 is a solid choice for browser automation when you need to interact with JavaScript-heavy pages or perform full user simulations. Key points to remember: use the By class for all element lookups, use WebDriverWait instead of sleep(), and always call driver.quit() when finished.

For simpler HTTP-based scraping where JavaScript rendering isn't required, consider requests + BeautifulSoup instead. For modern async browser automation, Playwright is a strong alternative with a cleaner API.