MultirotorClient().simGetImages() stops returning frames at some random point during the simulation

[maksym-borovyk]

Bug report

• AirSim Version/#commit: 1.8.1
• UE/Unity version: I am using custom binary of a map
• autopilot version: ArduCopter-4.5.7 (stab)
• OS Version: Ubuntu 20.04.6 LTS
• GPU: NVIDIA GeForce RTX 2080

What's the issue you encountered?

During 30-50% of experiments (~ 5 min flight) on a particular server, I encounter problem where the frame request is not answered at some random point and hangs indefinitely.

client = airsim.MultirotorClient()

Request code:

responses = client.simGetImages(
    [
        airsim.ImageRequest(
            camera_name, airsim.ImageType.Scene, False, False
        )
    ]
)

If I add a timeout like client = airsim.MultirotorClient(timeout_value=1) and retry requests after failure - new requests still don't return frames.
There are no new messages in AirSim terminal during such failures. It doesn't seem like a full AirSim failure, because I still can retrieve ground truth data from the simulation.

I can't replicate the problem on other servers; this issue only occurs on one specific server.

Resource utilization before the failure doesn't seem to approach server limits:

The drop in resource consumption after the failure in the plots above is at least partially due to a decrease in AirSim's resource usage after the failure.

Settings

How can the issue be reproduced?

Basically it can be something like

import airsim

client = airsim.MultirotorClient()

while True:
    responses = client.simGetImages(
        [
            airsim.ImageRequest(
                camera_name, airsim.ImageType.Scene, False, False
            )
        ]
    )

Unfortunately, I cannot share the map's binary.

Include full error message in text form

There is no error message. Frames just stop being returned at some point.

What's better than filing an issue? Filing a pull request :).