Asynchronous downloads using asyncio in Python

asyncio is especially useful for I/O-bound tasks like downloading or uploading files, where the primary bottleneck is neither CPU nor Memory but rather waiting for network responses.

asyncio is part of the standard lib for python3 but you need to install aiohttp by running pip install aiohttp.

1. Files to download

Our research project will attempt to download about 200k images of the Earth from the NASA Photo Journal.

urls = [
    "https://photojournal.jpl.nasa.gov/jpeg/PIA00074.jpg",
    "https://photojournal.jpl.nasa.gov/jpeg/PIA00076.jpg",
    .....
]

2. Prepare directory to save the files

DOWNLOAD_DIR = 'downloads'
os.makedirs(DOWNLOAD_DIR, exist_ok=True)

3. Async download function

async def download_file(session, url):
    filename = os.path.join(DOWNLOAD_DIR, os.path.basename(url))
    async with session.get(url) as response:
        content = await response.read()
        with open(filename, 'wb') as f:
            f.write(content)
        print(f'{filename} downloaded.')

aiohttp.ClientSession to handle HTTP requests asynchronously.
Read the response content asynchronously and write to a file.

4. Main function to manage the downloads

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [download_file(session, url) for url in urls]
        await asyncio.gather(*tasks)

Create a list of tasks for each URL and run them concurrently using asyncio.gather.

5. Running the main function

asyncio.run(main(urls))

downloads\PIA00076.jpg downloaded.
downloads\PIA00074.jpg downloaded.
downloads\PIA00114.jpg downloaded.

Note: Asyncio uses a single-threaded event loop and is therefore ideal for I/O-bound tasks. Threading however, uses multiple threads running in parallel making it suitable for CPU-bound tasks.

Improvements

We can consider a few improvements:

Ensure that aiohttp is using a connection pool to manage multiple concurrent connections efficiently.
Add error handling to manage cases where some downloads might fail due to intermittent issues. Perhaps add some retry logic also?
Use a semaphore to limit the number of concurrent downloads to avoid overwhelming the server and risk of DDoS.
Although writing the file to disk is typically fast, we can ensure it’s not blocking the event loop by using asynchronous file writing.
Finally, we can handle the return values from asyncio.gather to collect results or exceptions

import asyncio
import aiohttp
import aiofiles
import os
import logging
from aiohttp import ClientSession
from aiohttp_retry import RetryClient, ExponentialRetry

# Directory to save the downloaded files
DOWNLOAD_DIR = 'downloads'
os.makedirs(DOWNLOAD_DIR, exist_ok=True)

# Limit the number of concurrent downloads
CONCURRENT_DOWNLOADS = 10

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

async def download_file(session, url, semaphore):
    async with semaphore:
        filename = os.path.join(DOWNLOAD_DIR, os.path.basename(url))
        try:
            async with session.get(url) as response:
                if response.status == 200:
                    async with aiofiles.open(filename, 'wb') as f:
                        content = await response.read()
                        await f.write(content)
                    logger.info(f'{filename} downloaded.')
                else:
                    logger.error(f'Failed to download {url}, status code: {response.status}')
        except Exception as e:
            logger.error(f'Error downloading {url}: {e}')

async def main(urls):
    semaphore = asyncio.Semaphore(CONCURRENT_DOWNLOADS)
    retry_options = ExponentialRetry(attempts=5)
    async with RetryClient(raise_for_status=False, retry_options=retry_options) as session:
        tasks = [download_file(session, url, semaphore) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        for url, result in zip(urls, results):
            if isinstance(result, Exception):
                logger.error(f'Error occurred when downloading {url}: {result}')

# List of URLs to download
urls = [
    "https://photojournal.jpl.nasa.gov/jpeg/PIA00074.jpg",
    "https://photojournal.jpl.nasa.gov/jpeg/PIA00076.jpg",
    "https://photojournal.jpl.nasa.gov/jpeg/PIA00114.jpg",
]

# Run the main function
asyncio.run(main(urls))

aiohttp_retry is used to automatically retry failed requests.
ExponentialRetry sets up exponential backoff for retries, attempting up to 5 retries.
Logging is configured for better visibility of the process and errors.
Increased the number of concurrent downloads to 10 for potentially better performance.
asyncio.gather is used with return_exceptions=True to handle exceptions for each task. This allows the program to continue running even if some downloads fail.
Added logging to surface download success and failure information.
RetryClient ensures that transient errors are retried with exponential backoff to make for a robust app.

Improvements

Further Reading