Skip to main content

Python HTTP Requests

Introduction​

In the world of web development, communication between clients and servers happens through HTTP (Hypertext Transfer Protocol). When you browse a website, your browser sends HTTP requests to servers and receives HTTP responses. As a Python developer, you'll often need to make HTTP requests to interact with web services, consume APIs, or scrape websites.

In this tutorial, we'll explore how to make HTTP requests in Python using the popular requests library. We'll learn how to:

  • Send GET, POST, PUT, and DELETE requests
  • Handle request headers and parameters
  • Process JSON responses
  • Work with authentication
  • Handle errors and exceptions

By the end of this tutorial, you'll be able to interact with any web API using Python.

The Requests Library​

Python's requests library is a simple, elegant, and powerful HTTP client that makes sending HTTP requests remarkably easy. It abstracts away the complexities of making HTTP requests so you can focus on interacting with services and using the data in your application.

Installation​

Before we begin, let's install the requests library:

bash
pip install requests

Once installed, you can import it in your Python script:

python
import requests

Making GET Requests​

GET requests are used to retrieve information from a server. Let's start with a simple example:

python
import requests

# Send a GET request to a URL
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')

# Check if the request was successful
if response.status_code == 200:
    # Print the response content
    print(response.text)
else:
    print(f"Request failed with status code: {response.status_code}")

Output:

{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

Understanding the Response Object​

The response object contains all the information returned by the server:

python
# Status code
print(f"Status code: {response.status_code}")

# Response headers
print(f"Headers: {response.headers}")

# Content type
print(f"Content type: {response.headers['Content-Type']}")

# Response encoding
print(f"Encoding: {response.encoding}")

# Response content as bytes
print(f"Content (bytes): {response.content[:50]}...")

# Response content as text
print(f"Content (text): {response.text[:50]}...")

Output:

Status code: 200
Headers: {'Date': 'Mon, 20 Mar 2023 12:34:56 GMT', 'Content-Type': 'application/json; charset=utf-8', ...}
Content type: application/json; charset=utf-8
Encoding: utf-8
Content (bytes): b'{\n  "userId": 1,\n  "id": 1,\n  "title": "sunt aut '...
Content (text): {
  "userId": 1,
  "id": 1,
  "title": "sunt aut ...

Working with JSON Responses​

Many modern APIs return data in JSON format. The requests library makes it easy to work with JSON responses:

python
import requests

response = requests.get('https://jsonplaceholder.typicode.com/posts/1')

# Parse the JSON response
data = response.json()

# Access the data as a Python dictionary
print(f"Post title: {data['title']}")
print(f"Post body: {data['body']}")

Output:

Post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
Post body: quia et suscipit
suscipit recusandae consequuntur expedita et cum
reprehenderit molestiae ut ut quas totam
nostrum rerum est autem sunt rem eveniet architecto

URL Parameters​

You often need to pass parameters in your GET requests. You can do this by providing a dictionary to the params argument:

python
import requests

# Query parameters as a dictionary
params = {
    'userId': 1,
    'completed': 'false'
}

response = requests.get('https://jsonplaceholder.typicode.com/todos', params=params)

# Print the URL that was requested
print(f"Requested URL: {response.url}")

# Process the JSON response
todos = response.json()
print(f"Found {len(todos)} todos")
print("First 2 todos:")
for todo in todos[:2]:
    print(f"- {todo['title']}")

Output:

Requested URL: https://jsonplaceholder.typicode.com/todos?userId=1&completed=false
Found 10 todos
First 2 todos:
- delectus aut autem
- quis ut nam facilis et officia qui

Making POST Requests​

POST requests are used to send data to a server to create or update a resource. Here's how to send a POST request with JSON data:

python
import requests

# Data to send in the POST request
data = {
    'title': 'foo',
    'body': 'bar',
    'userId': 1
}

# Send a POST request with JSON data
response = requests.post('https://jsonplaceholder.typicode.com/posts', json=data)

# Check if the request was successful
if response.status_code == 201:  # 201 Created
    created_post = response.json()
    print(f"Post created with ID: {created_post['id']}")
    print(f"Title: {created_post['title']}")
else:
    print(f"Request failed with status code: {response.status_code}")

Output:

Post created with ID: 101
Title: foo

Sending Form Data​

You can also send form-encoded data using the data parameter:

python
import requests

# Form data to send
form_data = {
    'username': 'johndoe',
    'password': 'secret123'
}

# Send form data (application/x-www-form-urlencoded)
response = requests.post('https://httpbin.org/post', data=form_data)

# Print the response
print(response.json()['form'])

Output:

{'password': 'secret123', 'username': 'johndoe'}

Request Headers​

HTTP headers let the client and server pass additional information with the request or response. Here's how to set custom headers:

python
import requests

# Custom headers
headers = {
    'User-Agent': 'My Python App/1.0',
    'Accept': 'application/json',
    'Authorization': 'Bearer your_token_here'
}

response = requests.get('https://httpbin.org/headers', headers=headers)

# Print the headers received by the server
print(response.json()['headers'])

Output:

{
  "Accept": "application/json",
  "Authorization": "Bearer your_token_here",
  "Host": "httpbin.org",
  "User-Agent": "My Python App/1.0",
  ...
}

Authentication​

The requests library supports various authentication methods:

Basic Authentication​

python
import requests
from requests.auth import HTTPBasicAuth

response = requests.get(
    'https://httpbin.org/basic-auth/user/pass',
    auth=HTTPBasicAuth('user', 'pass')
)

# Check if authentication was successful
if response.status_code == 200:
    print("Authentication successful!")
    print(response.json())
else:
    print(f"Authentication failed with status code: {response.status_code}")

Output:

Authentication successful!
{'authenticated': True, 'user': 'user'}

Token Authentication​

python
import requests

# API token
token = "your_api_token_here"

# Method 1: Through headers
headers = {
    'Authorization': f'Bearer {token}'
}
response = requests.get('https://httpbin.org/headers', headers=headers)

# Method 2: Using auth parameter
from requests.auth import AuthBase

class TokenAuth(AuthBase):
    def __init__(self, token):
        self.token = token
        
    def __call__(self, r):
        r.headers['Authorization'] = f'Bearer {self.token}'
        return r

response = requests.get('https://httpbin.org/headers', auth=TokenAuth(token))

Making PUT and DELETE Requests​

PUT Request​

PUT requests are used to update existing resources:

python
import requests

# Data for updating a resource
data = {
    'id': 1,
    'title': 'Updated Title',
    'body': 'This post has been updated',
    'userId': 1
}

# Send PUT request
response = requests.put('https://jsonplaceholder.typicode.com/posts/1', json=data)

# Check if the update was successful
if response.status_code == 200:
    updated_post = response.json()
    print("Post updated successfully!")
    print(f"New title: {updated_post['title']}")
else:
    print(f"Update failed with status code: {response.status_code}")

Output:

Post updated successfully!
New title: Updated Title

DELETE Request​

DELETE requests are used to remove resources:

python
import requests

# Send DELETE request
response = requests.delete('https://jsonplaceholder.typicode.com/posts/1')

# Check if the delete was successful
if response.status_code == 200:
    print("Post deleted successfully!")
else:
    print(f"Delete failed with status code: {response.status_code}")

Output:

Post deleted successfully!

Error Handling​

When working with HTTP requests, it's important to handle errors properly:

python
import requests

try:
    # Try to access a non-existent URL
    response = requests.get('https://jsonplaceholder.typicode.com/nonexistent', timeout=5)
    
    # Raise an exception if the response contains an HTTP error
    response.raise_for_status()
    
    # Process the response if no exception was raised
    print("Request successful!")
    print(response.json())
    
except requests.exceptions.HTTPError as errh:
    print(f"HTTP Error: {errh}")
except requests.exceptions.ConnectionError as errc:
    print(f"Error Connecting: {errc}")
except requests.exceptions.Timeout as errt:
    print(f"Timeout Error: {errt}")
except requests.exceptions.RequestException as err:
    print(f"Something went wrong: {err}")

Output:

HTTP Error: 404 Client Error: Not Found for url: https://jsonplaceholder.typicode.com/nonexistent

Sessions​

If you need to make multiple requests to the same server, using a session can improve performance by reusing the same TCP connection:

python
import requests

# Create a session
session = requests.Session()

# Configure the session with default headers
session.headers.update({
    'User-Agent': 'My Python App/1.0',
    'Accept': 'application/json'
})

# Make requests using the session
response1 = session.get('https://jsonplaceholder.typicode.com/posts/1')
response2 = session.get('https://jsonplaceholder.typicode.com/posts/2')

print(f"First post title: {response1.json()['title']}")
print(f"Second post title: {response2.json()['title']}")

# Close the session when you're done
session.close()

Output:

First post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
Second post title: qui est esse

Real-World Example: Working with a Weather API​

Let's create a practical example using a free weather API to get the current weather for a city:

python
import requests
import time

def get_weather(city, api_key="your_openweathermap_api_key"):
    """
    Get the current weather for a city using the OpenWeatherMap API
    
    For this example, if you don't have an API key, you can use the following:
    - Use the sample response below
    - Sign up for a free API key at https://openweathermap.org/api
    """
    # Comment out the real API call and use a sample response if you don't have an API key
    """
    base_url = "https://api.openweathermap.org/data/2.5/weather"
    params = {
        "q": city,
        "appid": api_key,
        "units": "metric"  # Use metric units (Celsius)
    }
    
    response = requests.get(base_url, params=params)
    response.raise_for_status()  # Raise an exception for HTTP errors
    
    return response.json()
    """
    
    # Sample response for demonstration purposes
    time.sleep(1)  # Simulate API request delay
    return {
        "name": city,
        "main": {
            "temp": 22.5,
            "feels_like": 21.8,
            "temp_min": 20.2,
            "temp_max": 24.3,
            "humidity": 65
        },
        "weather": [
            {
                "main": "Clear",
                "description": "clear sky"
            }
        ],
        "wind": {
            "speed": 3.6
        }
    }

def display_weather(weather_data):
    """Format and display the weather data"""
    print(f"\nWeather in {weather_data['name']}:")
    print(f"Temperature: {weather_data['main']['temp']}°C")
    print(f"Feels like: {weather_data['main']['feels_like']}°C")
    print(f"Conditions: {weather_data['weather'][0]['description']}")
    print(f"Humidity: {weather_data['main']['humidity']}%")
    print(f"Wind speed: {weather_data['wind']['speed']} m/s")

# Get weather for different cities
cities = ["London", "New York", "Tokyo", "Sydney"]

for city in cities:
    try:
        print(f"Getting weather data for {city}...")
        weather_data = get_weather(city)
        display_weather(weather_data)
    except requests.exceptions.RequestException as e:
        print(f"Error getting weather for {city}: {e}")

Output:

Getting weather data for London...

Weather in London:
Temperature: 22.5°C
Feels like: 21.8°C
Conditions: clear sky
Humidity: 65%
Wind speed: 3.6 m/s

Getting weather data for New York...

Weather in New York:
Temperature: 22.5°C
Feels like: 21.8°C
Conditions: clear sky
Humidity: 65%
Wind speed: 3.6 m/s

Getting weather data for Tokyo...

Weather in Tokyo:
Temperature: 22.5°C
Feels like: 21.8°C
Conditions: clear sky
Humidity: 65%
Wind speed: 3.6 m/s

Getting weather data for Sydney...

Weather in Sydney:
Temperature: 22.5°C
Feels like: 21.8°C
Conditions: clear sky
Humidity: 65%
Wind speed: 3.6 m/s

Summary​

In this tutorial, you've learned how to:

  1. Make HTTP GET, POST, PUT, and DELETE requests using the Python requests library
  2. Process JSON responses and handle request parameters
  3. Set custom headers and handle authentication
  4. Handle errors and exceptions
  5. Use sessions for improved performance
  6. Apply these concepts to a real-world example with a weather API

Python's requests library makes it easy to interact with web services and APIs, opening up a world of possibilities for your applications. Whether you're building a web scraper, integrating with third-party services, or developing your own web applications, understanding HTTP requests is a fundamental skill for any Python web developer.

Additional Resources​

Here are some resources to learn more about Python HTTP requests:

  1. Requests: HTTP for Humans™ - Official documentation
  2. HTTP Status Codes - Mozilla Developer Network
  3. RESTful API Design - Best practices for designing RESTful APIs

Exercises​

To practice what you've learned:

  1. Create a simple script that retrieves and displays the latest posts from a subreddit using the Reddit API
  2. Build a command-line tool that fetches and displays the current exchange rate between two currencies
  3. Create a function that checks if a website is online by making an HTTP request and checking the status code
  4. Write a script that downloads a file from a given URL and shows a progress bar
  5. Build a simple web scraper that extracts article titles and links from a news website

Happy coding!


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)