Python: Selenium Search di Google
Imagine what would you do if you could automate all the repetitive and boring activities you perform using internet, like checking every day the first results of Google for a given keyword, or download a bunch of files from different websites.
In this post you’ll learn to use Selenium with Python, a Web Scraping tool that simulates a user surfing the Internet. For example, you can use it to automatically look for Google queries and read the results, log in to your social accounts, simulate a user to test your web application, and anything you find in your daily live that it’s repetitive. The possibilities are infinite! 🙂
- All the code in this post has been tested with Python 2.7 and Python 3.4.
Install and use Selenium
Selenium is a python package that can be installed via pip. I recommend that you install it in a virtual environment (using virtualenv and virtualenvwrapper). No virtualenv or virtualenvwrapper?
Learn how to create one here, for Python 2.7 and for Python 3. It’s really useful, once you start using them you won’t stop! 🙂
Remember that to create the environment in Python 2.7, just type: $ mkvirtualenv selenium_env 1
$ mkvirtualenv selenium_env
and in Python 3: $ mkvirtualenv --python=/usr/local/bin/python3 selenium_env 1
$ mkvirtualenv --python=/usr/local/bin/python3 selenium_env
where you should use your own Python 3 path.
Note: if you don’t want to use a virtual environment, you can still install the packages directly on you computer.
To install selenium, you just need to type: $ pip install selenium 1
$ pip install selenium
In this post we are going to initialize a Firefox driver — you can install it by visiting their website. However, if you want to work with Chrome or IE, you can find more information here.
Once you have Selenium and Firefox installed, create a python file, selenium_script.py. We are going to initialize a browser using Selenium: import time from selenium import webdriver
driver = webdriver.Firefox() time.sleep(5) driver.quit() 1 2 3 4 5 6
import time from selenium import webdriver
driver = webdriver.Firefox() time.sleep(5) driver.quit()
This just initializes a Firefox instance, waits for 5 seconds, and closes it.
Well, that was not very useful…
How about if we go to Google and search for something? Web Scraping Google with Selenium
Let’s make a script that loads the main Google search page and makes a query to look for “Selenium”: import time from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException
def init_driver():
driver = webdriver.Firefox() driver.wait = WebDriverWait(driver, 5) return driver
def lookup(driver, query):
driver.get("http://www.google.com")
try:
box = driver.wait.until(EC.presence_of_element_located(
(By.NAME, "q")))
button = driver.wait.until(EC.element_to_be_clickable(
(By.NAME, "btnK")))
box.send_keys(query)
button.click()
except TimeoutException:
print("Box or Button not found in google.com")
if __name__ == "__main__":
driver = init_driver() lookup(driver, "Selenium") time.sleep(5) driver.quit()
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
import time from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException
def init_driver():
driver = webdriver.Firefox() driver.wait = WebDriverWait(driver, 5) return driver
def lookup(driver, query):
driver.get("http://www.google.com")
try:
box = driver.wait.until(EC.presence_of_element_located(
(By.NAME, "q")))
button = driver.wait.until(EC.element_to_be_clickable(
(By.NAME, "btnK")))
box.send_keys(query)
button.click()
except TimeoutException:
print("Box or Button not found in google.com")
if __name__ == "__main__":
driver = init_driver() lookup(driver, "Selenium") time.sleep(5) driver.quit()
In the previous code:
the function init_driver initializes a driver instance.
creates the driver instance
adds the WebDriverWait function as an attribute to the driver, so it can be accessed more easily. This function is used to make the driver wait a certain amount of time (here 5 seconds) for an event to occur.
the function lookup takes two arguments: a driver instance and a query lookup (a string).
it loads the Google search page
it waits for the query box element to be located and for the button to be clickable. Note that we are using the WebDriverWait function to wait for these elements to appear.
Both elements are located by name. Other options would be to locate them by ID, XPATH, TAG_NAME, CLASS_NAME, CSS_SELECTOR , etc (see table below). You can find more information here.
Next, it sends the query into the box element and clicks the search button.
If either the box or button are not located during the time established in the wait function (here, 5 seconds), the TimeoutException is raised.
the next statement is a conditional that is true only when the script is run directly. This prevents the next statements to run when this file is imported.
it initializes the driver and calls the lookup function to look for “Selenium”.
it waits for 5 seconds to see the results and quits the driver
Finally, run your code with: $ python selenium_script.py 1
$ python selenium_script.py
Did it work? If you got an ElementNotVisibleException , keep reading!