【Python】aiohttp raise RuntimeError('Event loop is closed')
2022年 09月 28 日

子弹

问题描述

aiohttp 的 getting started 入门案例是这样写的

import aiohttp
import asyncio


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://python.org') as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

运行结果为

Status: 200
Content-type: text/html; charset=utf-8
Body: <!doctype html> ...

看上去没问题,但是在 Python3.7 后对 asyncio 进行了改进,可以直接调用 asyncio.run() 执行协程程序,而不需管底层 API,如事件循环 loop 的操作,所以上述代码的

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

可以直接替换为

asyncio.run(main())

Linux 和 Mac 上这样运行是没问题的,但是在 Windows 上运行会报如下错误

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001B1FFE978B0>
Traceback (most recent call last):
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

原因分析

像 aiohttp 这类第三方协程库都是依赖于标准库 asyncio 的,而 asyncio 对 Windows 的支持本来就不好。Python3.8 后默认 Windows 系统上的事件循环采用 ProactorEventLoop(仅用于 Windows)

这篇文档描述了其在 Windows 下的缺陷:

https://docs.python.org/zh-cn/3/library/asyncio-platforms.html#windows

引发异常的函数是 _ProactorBasePipeTransport.__del__ ,所以 aiohttp 铁定使用了 _ProactorBasePipeTransport,并且在程序退出释放内存时自动调用了其__del__ 方法



就是上述一串连环反应最终抛出了 RuntimeError: Event loop is closed

一般的协程程序是不会使用 _ProactorBasePipeTransport 的,所以下面的代码还是可以正常的运行

import asyncio


async def main():
    print('Hello...')
    await asyncio.sleep(1)
    print('...World')


asyncio.run(main())

我特意写了个装饰器来验证这一点:

import asyncio
from asyncio.proactor_events import _ProactorBasePipeTransport
from functools import wraps


def explicit_call(func):
    @wraps(func)
    def wrappers(self, *args, **kwargs):
        print('call _ProactorBasePipeTransport.__del__')
        return func(self, *args, **kwargs)

    return wrappers


_ProactorBasePipeTransport.__del__ = explicit_call(_ProactorBasePipeTransport.__del__)


async def main():
    print('Hello...')
    await asyncio.sleep(1)
    print('...World')


asyncio.run(main())

正常执行,没有使用 _ProactorBasePipeTransport

Hello...
...World
import asyncio
from asyncio.proactor_events import _ProactorBasePipeTransport
from functools import wraps

import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.baidu.com') as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")


def explicit_call(func):
    @wraps(func)
    def wrappers(self, *args, **kwargs):
        print('call _ProactorBasePipeTransport.__del__')
        return func(self, *args, **kwargs)

    return wrappers


_ProactorBasePipeTransport.__del__ = explicit_call(_ProactorBasePipeTransport.__del__)

asyncio.run(main())

先打印 call _ProactorBasePipeTransport.__del__ 然后报错,说明使用了 _ProactorBasePipeTransport

Status: 200
Content-type: text/html
Body: <html>
 ...
call _ProactorBasePipeTransport.__del__
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001B11653EF70>
Traceback (most recent call last):
  File "D:\Codes\SpiderProject\BingImageSpider\demo.py", line 16, in wrappers
    return func(self, *args, **kwargs)
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

进程已结束,退出代码为 0

解决方案

如果执意要在 Windows 下继续开发,有这几个方案可以选择

1. 不要使用 run 函数

既然 _ProactorBasePipeTransport 会在程序结束后自动关闭事件循环,那就不要用 run 函数了,用官网的例子,乖乖使用 loop 吧

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

2. 替换事件循环

在调用 run 函数前,替换默认的 ProactorEventLoop 为 SelectorEventLoop

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(main())

但是 SelectorEventLoop 是有一些缺点的,比如不支持子进程等

3. 忽略异常

这是 Github 上一个外国大佬的方法,在不改变源码的前提下,使用装饰器忽略掉异常

import asyncio
from asyncio.proactor_events import _ProactorBasePipeTransport
from functools import wraps

import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.baidu.com') as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")


def silence_event_loop_closed(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        try:
            return func(self, *args, **kwargs)
        except RuntimeError as e:
            if str(e) != 'Event loop is closed':
                raise

    return wrapper


_ProactorBasePipeTransport.__del__ = silence_event_loop_closed(_ProactorBasePipeTransport.__del__)

asyncio.run(main())

更详细的信息可以在这个 issue 上找到:https://github.com/aio-libs/aiohttp/issues/4324

【Python】aiohttp raise RuntimeError('Event loop is closed')

问题描述

aiohttp 的 getting started 入门案例是这样写的

import aiohttp
import asyncio


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://python.org') as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

运行结果为

Status: 200
Content-type: text/html; charset=utf-8
Body: <!doctype html> ...

看上去没问题,但是在 Python3.7 后对 asyncio 进行了改进,可以直接调用 asyncio.run() 执行协程程序,而不需管底层 API,如事件循环 loop 的操作,所以上述代码的

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

可以直接替换为

asyncio.run(main())

Linux 和 Mac 上这样运行是没问题的,但是在 Windows 上运行会报如下错误

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001B1FFE978B0>
Traceback (most recent call last):
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

原因分析

像 aiohttp 这类第三方协程库都是依赖于标准库 asyncio 的,而 asyncio 对 Windows 的支持本来就不好。Python3.8 后默认 Windows 系统上的事件循环采用 ProactorEventLoop(仅用于 Windows)

这篇文档描述了其在 Windows 下的缺陷:

https://docs.python.org/zh-cn/3/library/asyncio-platforms.html#windows

引发异常的函数是 _ProactorBasePipeTransport.__del__ ,所以 aiohttp 铁定使用了 _ProactorBasePipeTransport,并且在程序退出释放内存时自动调用了其__del__ 方法



就是上述一串连环反应最终抛出了 RuntimeError: Event loop is closed

一般的协程程序是不会使用 _ProactorBasePipeTransport 的,所以下面的代码还是可以正常的运行

import asyncio


async def main():
    print('Hello...')
    await asyncio.sleep(1)
    print('...World')


asyncio.run(main())

我特意写了个装饰器来验证这一点:

import asyncio
from asyncio.proactor_events import _ProactorBasePipeTransport
from functools import wraps


def explicit_call(func):
    @wraps(func)
    def wrappers(self, *args, **kwargs):
        print('call _ProactorBasePipeTransport.__del__')
        return func(self, *args, **kwargs)

    return wrappers


_ProactorBasePipeTransport.__del__ = explicit_call(_ProactorBasePipeTransport.__del__)


async def main():
    print('Hello...')
    await asyncio.sleep(1)
    print('...World')


asyncio.run(main())

正常执行,没有使用 _ProactorBasePipeTransport

Hello...
...World
import asyncio
from asyncio.proactor_events import _ProactorBasePipeTransport
from functools import wraps

import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.baidu.com') as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")


def explicit_call(func):
    @wraps(func)
    def wrappers(self, *args, **kwargs):
        print('call _ProactorBasePipeTransport.__del__')
        return func(self, *args, **kwargs)

    return wrappers


_ProactorBasePipeTransport.__del__ = explicit_call(_ProactorBasePipeTransport.__del__)

asyncio.run(main())

先打印 call _ProactorBasePipeTransport.__del__ 然后报错,说明使用了 _ProactorBasePipeTransport

Status: 200
Content-type: text/html
Body: <html>
 ...
call _ProactorBasePipeTransport.__del__
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001B11653EF70>
Traceback (most recent call last):
  File "D:\Codes\SpiderProject\BingImageSpider\demo.py", line 16, in wrappers
    return func(self, *args, **kwargs)
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\DevTools\Python\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "D:\DevTools\Python\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

进程已结束,退出代码为 0

解决方案

如果执意要在 Windows 下继续开发,有这几个方案可以选择

1. 不要使用 run 函数

既然 _ProactorBasePipeTransport 会在程序结束后自动关闭事件循环,那就不要用 run 函数了,用官网的例子,乖乖使用 loop 吧

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

2. 替换事件循环

在调用 run 函数前,替换默认的 ProactorEventLoop 为 SelectorEventLoop

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(main())

但是 SelectorEventLoop 是有一些缺点的,比如不支持子进程等

3. 忽略异常

这是 Github 上一个外国大佬的方法,在不改变源码的前提下,使用装饰器忽略掉异常

import asyncio
from asyncio.proactor_events import _ProactorBasePipeTransport
from functools import wraps

import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.baidu.com') as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")


def silence_event_loop_closed(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        try:
            return func(self, *args, **kwargs)
        except RuntimeError as e:
            if str(e) != 'Event loop is closed':
                raise

    return wrapper


_ProactorBasePipeTransport.__del__ = silence_event_loop_closed(_ProactorBasePipeTransport.__del__)

asyncio.run(main())

更详细的信息可以在这个 issue 上找到:https://github.com/aio-libs/aiohttp/issues/4324

赞 (3)

评论区(暂无评论)

这里空空如也,快来评论吧~

我要评论