Python has been a popular programming language for a while now. It’s high-level, easy-to-learn, and portable nature has served a wide array of applications. Over the years, Python developers have relied on third-party libraries and packages to build powerful and scalable applications. One such library is “asyncio”.
Asyncio, also called “Tulip” in its inception, is a Python package for writing concurrent code using the async/await syntax. It’s part of the standard library since Python 3.4, and it inspired the development of many useful libraries such as aiohttp, asyncpg, and aioredis.
Asyncio allows us to write millions of networked clients and servers with little effort while maintaining high performance and low memory requirements.
One of the critical features of asyncio is that it enables the development of non-blocking code that can handle multiple operations simultaneously. This feature allows programmers to write code that can work in streams, so they don’t have to wait for the end of a process before starting another.
Asyncio’s gathering functionality achieves this by enabling the creation of multiple concurrent tasks and waiting for them to complete in a non-blocking way.
In this article, we will explore asyncio gather, its use cases, and how to use it with code examples.
What is asyncio gather?
Asyncio gather is a function that enables multiple functions to run concurrently and waits for them to complete before returning the result.
Asyncio provides three primary functions for running tasks concurrently: asyncio.gather(), asyncio.wait(), and asyncio.as_completed(). In this article, we focus on asyncio.gather().
According to the official documentation, asyncio gather function “runs awaitable objects in the provided iterable concurrently and blocks until the results are available.”
Asyncio gather takes in an iterable of coroutine objects, scheduled them to run concurrently, and wait for them to complete.
Code example:
Let’s create a simple program that uses asyncio gather to perform two concurrent tasks;
- The first task simulates sending an email
- The second task simulates saving a file.
import asyncio
async def email_sender():
print(f"Sending email...")
await asyncio.sleep(2)
print(f"Email sent!")
async def file_saver():
print(f"Saving file...")
await asyncio.sleep(3)
print(f"File saved!")
async def main():
tasks = [email_sender(), file_saver()]
await asyncio.gather(*tasks)
if __name__ == "__main__":
asyncio.run(main())
Output:
Sending email...
Saving file...
Email sent!
File saved!
The program executes the email_sender and file_saver tasks concurrently using the asyncio gather function. The gather function then waits for both tasks to complete before the program proceeds.
From the output, we can see that the program prints the message for both tasks before continuing with the rest of the program.
Asyncio gather has a few critical features that we need to discuss.
- Running tasks concurrently
Asyncio gather enables us to run multiple tasks concurrently, which means that tasks will be executed simultaneously. When a task becomes blocked, asyncio scheduler switches to a different task and only switches back when the blocked task is done.
tasks = [task_1(), task_2(), task_3()]
await asyncio.gather(*tasks)
- Gathering results
Another useful feature of asyncio gather is that it gathers the results of the tasks and returns them in the order of the tasks provided. You can retrieve the results using the “return” statement.
tasks = [task_1(), task_2(), task_3()]
results = await asyncio.gather(*tasks)
- Error handling
If one of the tasks raises an exception or errors out, asyncio gather will cancel all other tasks and return the exception immediately.
async def task_1():
raise ValueError("Exception in task_1")
async def task_2():
print("Task 2 is running...")
async def main():
tasks = [task_1(), task_2()]
results = await asyncio.gather(*tasks, return_exceptions=True)
print(results)
if __name__ == "__main__":
asyncio.run(main())
Output:
[ValueError('Exception in task_1'), None]
As we can see from the output, the tasks stopped running immediately after the first task raised the ValueError exception. However, the remaining task still returned None.
4. Cancelling tasks
Asyncio gather tasks can be cancelled by calling the “Task.cancel” method.
async def task_1():
print("Task 1 is running…")
await asyncio.sleep(3)
print("Task 1 done!")
async def task_2():
print("Task 2 is running…")
await asyncio.sleep(4)
print("Task 2 done!")
async def main():
task_1_res = asyncio.create_task(task_1())
task_2_res = asyncio.create_task(task_2())
# Wait for 2 seconds before cancelling the tasks
await asyncio.sleep(2)
task_1_res.cancel()
task_2_res.cancel()
results = await asyncio.gather(task_1_res, task_2_res, return_exceptions=True)
print(results)
if name == "main":
asyncio.run(main())
Output:
Task 2 is running…
Task 1 done!
The output shows that the second task continued running after the first task was cancelled.
Note: for this, we used asyncio.create_task instead of directly passing coroutines to asyncio.gather function. If we do not use asyncio.create_task, this may cause an error while cancelling the task.
Conclusion:
Asyncio gather is a powerful tool for building scalable, high-performance applications. It allows developers to write non-blocking, concurrent code using the async/await syntax and perform multiple tasks simultaneously.
In this article, we discussed what asyncio gather is, its features, and how to use it effectively with code examples. We hope that this article has improved your understanding of asyncio gather and its potential uses in your projects.
let's dive deeper into some of the topics we discussed in the previous section.
Concurrency vs Parallelism
Concurrency and parallelism are two concepts that are often used interchangeably, but they refer to different things.
Concurrency is the ability of a system to do multiple things at the same time, even if it’s using the same resources. It enables a single processor to switch between multiple tasks, giving the impression of simultaneous execution.
Parallelism, on the other hand, is when multiple processors work together to execute tasks, leading to true simultaneous execution.
In Python, concurrency is typically achieved using asynchronous programming, while parallelism is implemented using multi-processing.
In our example, asyncio gather uses concurrency to run multiple tasks concurrently, sharing the same event loop. This approach is especially useful for IO-bound tasks, where the application spends most of its time waiting for input/output operations to complete.
Coroutine Functions
Coroutine functions in Python are functions that can be paused and resumed during execution. They provide a mechanism for non-blocking, cooperative multitasking.
A coroutine function defined using the “async” keyword and executed using the “await” keyword. Coroutine functions can yield control to other coroutines while waiting for input/output operations to complete or other tasks to be processed.
Coroutines are lightweight and can run concurrently, making them ideal for IO-bound tasks.
In our example, we defined the email_sender and file_saver functions as coroutine functions using the “async” keyword.
async def email_sender():
print(f"Sending email…")
await asyncio.sleep(2)
print(f"Email sent!")
Asyncio Sleep
Asyncio sleep is a utility function that pauses the execution of the coroutine for a specified number of seconds. It does not block the entire event loop, allowing other coroutines to run in the meantime.
In our example, we used asyncio sleep to simulate the time taken to send an email and save a file.
async def email_sender():
print(f"Sending email…")
await asyncio.sleep(2)
print(f"Email sent!")
Asyncio Create_Task
In Python, we can create a task using asyncio.create_task() instead of executing the coroutine directly.
The asyncio create_task() function creates a task from a coroutine object and returns it. This makes it easier to manage various tasks and provides a standard way of cancelling tasks.
In our example, we used asyncio.create_task to create tasks for the email_sender() and file_saver() functions and run them concurrently.
task_1_res = asyncio.create_task(task_1())
task_2_res = asyncio.create_task(task_2())
Asyncio Cancel
Sometimes, we may want to cancel one or more tasks before they complete. To cancel a task in asyncio, we call the “Task.cancel()” method. This method raises the asyncio.CancelledError exception, which can be caught to handle cancelled tasks.
In our example, we cancelled the tasks after two seconds using the “Task.cancel()” method.
task_1_res.cancel()
task_2_res.cancel()
Parallelism with Multi-Processing
Unlike concurrency, which uses a single processor to switch between tasks, parallelism involves using multiple processors to execute tasks at the same time.
In Python, multiprocessing is used to achieve parallelism. The multiprocessing module enables the creation of subprocesses to distribute work across multiple cores, allowing us to execute code in parallel.
Conclusion:
In this article, we explored asyncio gather, a powerful tool for building scalable, high-performance applications in Python. We discussed concepts like concurrency, parallelism, coroutine functions, asyncio sleep, asyncio create_task, and asyncio cancel. We also looked at how multiprocessing can be used for parallelism in Python.
By leveraging the capabilities of asyncio gather, developers can write code that is responsive, scalable, and efficient, making it ideal for building complex applications and systems. We hope that this article has expanded your understanding of Python asyncio gather and its practical use cases.
## Popular questions
1. What is Python asyncio gather?
- Python asyncio gather is a function that allows multiple coroutine functions to run concurrently and waits for them to complete before returning the result.
2. How is concurrency different from parallelism in Python?
- In Python, concurrency involves a single processor switching between tasks while parallelism involves multiple processors executing tasks simultaneously.
3. How do you define coroutine functions in Python, and why are they useful?
- Coroutine functions in Python are defined using the "async" keyword, and they can be paused and resumed during execution. They provide a mechanism for non-blocking, cooperative multitasking.
4. How do you cancel tasks in asyncio, and why might you want to do so?
- To cancel a task in asyncio, you call the "Task.cancel()" method. You might want to cancel tasks if they are taking too long or if they are no longer needed.
5. Can you explain the difference between the asyncio "gather" and "wait" functions?
- The asyncio "gather" function runs all the provided coroutines concurrently and waits for them to complete before returning the results, while the "wait" function returns a set of completed tasks and incomplete ones. The "wait" function does not necessarily wait for all tasks to complete before returning.
### Tag
'AsyncioGather'
