Making a request in Python

Let's look at an example usage of the module. If you already have the code for this book downloaded from the GitHub page, go ahead and navigate to the Chapter05 folder. Let's take a look at the example1.py file, as shown in the following code:

# Chapter05/example1.py

import requests

url = 'http://www.google.com'

res = requests.get(url)

print(res.status_code)
print(res.headers)

with open('google.html', 'w') as f:
    f.write(res.text)

print('Done.')

In this example, we are using the requests module to download the HTML code of the web page, www.google.com. The requests.get() method sends a GET request method to url and we store the response to the res variable. After checking the status and headers of the response by printing them out, we create a file called google.html and write the HTML code, which is stored in the response text, to the file.

After running the programming (assuming that your internet is working and the Google server is not down), you should get the following output:

200
{'Date': 'Sat, 17 Nov 2018 23:08:58 GMT', 'Expires': '-1', 'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/html; charset=ISO-8859-1', 'P3P': 'CP="This is not a P3P policy! See g.co/p3phelp for more info."', 'X-XSS-Protection': '1; mode=block', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Encoding': 'gzip', 'Server': 'gws', 'Content-Length': '4958', 'Set-Cookie': '1P_JAR=2018-11-17-23; expires=Mon, 17-Dec-2018 23:08:58 GMT; path=/; domain=.google.com, NID=146=NHT7fic3mjBO_vdiFB3-gqnFPyGN1EGxyMkkNPnFMEVsqjGJ8S0EwrivDBWBgUS7hCPZGHbosLE4uxz31shnr3X4adRpe7uICEiK8qh3Asu6LH_bIKSLWStAp8gMK1f9_GnQ0_JKQoMvG-OLrT_fwV0hwTR5r2UVYsUJ6xHtX2s; expires=Sun, 19-May-2019 23:08:58 GMT; path=/; domain=.google.com; HttpOnly'}
Done.

The response had a 200 status code, which we know means that the request has been successfully completed. The header of the response, stored in res.headers, additionally contains further specific information regarding the response. For example, we can see the date and time the request was made or that the content of the response is text and HTML and the total length of the content is 4958.

The complete data sent from the server was also written to the google.html file. When you open the file in a text editor, you will be able to see the HTML code of the web page that we have downloaded using requests. On the other hand, if you use a web browser to open the file, you will see how most of the information from the original web page is now being displayed through a downloaded offline file.

For example, the following is how Google Chrome on my system interprets the HTML file:

There is other information that is stored on the server that web pages of that server make reference to. This means that not all of the information that an online web page provides can be downloaded via a GET request, and this is why offline HTML code sometimes fails to contain all of the information available on the online web page that it was downloaded from. (For example, the downloaded HTML code in the preceding screenshot does not display the Google icon correctly.)