Pythonistas, Read This If You Want to Start Extracting Data from Websites
The internet is more than just cat memes and viral videos — it’s an invaluable source of data that you can use to make better business decisions.
However, extracting data from websites isn’t always easy (or legal). If you’re willing to bend the rules, Python may be able to help you get the data you need to run your business better. Here are five ways Python makes web scraping easier.
Find the URL
- Find the URL
- Download the HTML code of the website you want to extract data from by right-clicking on the site and selecting View Page Source or View Source. Copy and paste this into a text document.
- Run your Python script with Python 3.5+.
- Your script is a .py file that you can use in any programming language.
- To save your program as a .py file, click File > Save as > Choose Save as type: Python 3.5+
- Make sure that the title of your python script matches the title of your blog post without spaces or punctuation in between words.
Send a GET Request
You can use python’s library urllib2 to send a GET request. The first step is to import the module by typing import urllib2 at the top of your Python script. After that you need to open a connection using the urlopen() function. Note that this function takes an URL as its argument.
Check Status Code
How do I check the status code of a website?
The quickest way to determine the status code of a website is by using your web browser.
In Google Chrome, for example, go up to the top right corner and click on Inspect. Underneath Network, you should see a series of numbers. The first one will be the status code. Status codes are typically three numbers that indicate what happened when you visited the site.
Find the Data
Many data extraction tools are available that work with Python. If you’re looking for a tool that can extract data in a variety of formats and upload the output to a specified location, try Scrapy.
Scrapy is an open-source framework built on Twisted and it has been used in many production sites. For more information about using Scrapy, check out A Gentle Introduction To Scrapy
Parse the HTML
You can use Python to extract the content of webpages without needing to do a lot of HTML and JavaScript parsing.
Enter Beautiful Soup. Beautiful Soup is an excellent library that has the ability to parse HTML pages into more usable data like text, links and lists of tags. Once you have these parsed out you can analyze the data with other Python libraries or create new analysis tools that are specific for your needs.
Save the Data
A lot of websites provide data in the form of API’s. API stands for Application Programming Interface and is a way for programmers to access content on websites.
There are many sites that provide their API’s publicly for anyone to use. Some common web resources that have API’s are Facebook and Twitter.
Conclusion
This is the theretical way of elaborating extracting data from a website using python.
