Estimate relative positions using onboard images

[LuciusLuF]

Project package: [https://sites.google.com/view/kavehfathian/code/vision-based-formation-control-in-unreal-engine-4-airsim
]
Thanks in advance for your time! I was following the instruction of Vision-Based Formation Control in Unreal Engine 4 - AirSim and the code was stuck at estimating relative positions using onboard images.

The error message is:
MatlabExecutionError:
"File C:\Users\90718\Desktop\Vision-Based Formation Control in Unreal Engine 4 - AirSim\GetRelativePose_Ver1_7.m, line 57, in GetRelativePose_Ver1_7
Index exceeds the number of array elements (0)."

The error code containing GetRelativePose_Ver1_7 is:

# Estimate relative positions using onboard images    
for i in range(numUAV):
    name = "UAV" + str(i+1)
    imgs = client.simGetImages([
        airsim.ImageRequest("0", airsim.ImageType.Scene, False, False)], vehicle_name = name)  # Scene vision image in uncompressed RGBA array
    img = imgs[0]
    if i == 0:
        imgArray = img.image_data_uint8
    else:
        imgArray = imgArray + img.image_data_uint8

# meng.edit('GetRelativePose_Ver1_7')
Qm, Tm, flagm = meng.GetRelativePose_Ver1_7(imgArray, imgWidth, imgHeight, save, Adjm, itr, nargout=3)
T = np.asarray(Tm)
flag = np.asarray(flagm).flatten()

imgArray is a variable of bytes type. Somehow it used Matlab engine to deal with the imgArray. So I went to the Matlab function and the error was located at:

imgs(:,:,1,i) = reshape( imgArray((i-1)*4*arrayLength+1 : 4 : i*4*arrayLength), imgWidth, imgHeight).';

The imgArray looks like:

The Matlab code looks like:

Is it because the bytes variable is not callable? Or the size of the matrix can't match?

[rajat2004]

The problem might be that it expects the image to be RGBA, but AirSim currently (and for 1-2 years I believe) provides RGB images

[LuciusLuF]

That makes sense. This code is written in 2018.

[yingjiewx]

hey,Have you been able to successfully run it now？I also want to run his code，I was stuck at estimating relative positions using onboard images too,but the mistake is not quite the same as yours.If you successfully run it, can you tell me exactly where the problem is with me?My environment is built entirely according to instructions.
error:
Traceback (most recent call last):

File "", line 1, in
runfile('E:/UE4_AirSim_Simulation/Simulation/FormationControl.py', wdir='E:/UE4_AirSim_Simulation/Simulation')

File "E:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

File "E:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "E:/UE4_AirSim_Simulation/Simulation/FormationControl.py", line 180, in
Qm, Tm, flagm = meng.GetRelativePose_Ver1_7(imgArray, imgWidth, imgHeight, save, Adjm, itr, nargout=3)

File "E:\Anaconda3\lib\site-packages\matlab\engine\matlabengine.py", line 71, in call
_stderr, feval=True).result()

File "E:\Anaconda3\lib\site-packages\matlab\engine\futureresult.py", line 67, in result
return self.__future.result(timeout)

File "E:\Anaconda3\lib\site-packages\matlab\engine\fevalfuture.py", line 82, in result
self._result = pythonengine.getFEvalResult(self._future,self._nargout, None, out=self._out, err=self._err)

MatlabExecutionError:
File E:\UE4_AirSim_Simulation\Simulation\QuEst\Helpers\FindTransDepth_Ver1_0.m, line 63, in FindTransDepth_Ver1_0

File E:\UE4_AirSim_Simulation\Simulation\QuEst\QuEst_Ver1_1.m, line 68, in QuEst_Ver1_1

File E:\UE4_AirSim_Simulation\Simulation\QuEst RANSAC\QuEst_RANSAC_Ver1_2.m, line 191, in PoseEstimator

File E:\UE4_AirSim_Simulation\Simulation\QuEst RANSAC\QuEst_RANSAC_Ver1_2.m, line 408, in ransac

File E:\UE4_AirSim_Simulation\Simulation\QuEst RANSAC\QuEst_RANSAC_Ver1_2.m, line 80, in QuEst_RANSAC_Ver1_2

File E:\UE4_AirSim_Simulation\Simulation\GetRelativePose_Ver1_7.m, line 164, in GetRelativePose_Ver1_7
using svd Input to SVD must not contain NaN or Inf.

[LuciusLuF]

zaowangle😭

…

---Original--- From: ***@***.***> Date: Sat, Apr 22, 2023 00:54 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [microsoft/AirSim] Estimate relative positions using onboardimages (Discussion #3503) hey,Have you been able to successfully run it now？I also want to run his code，I was stuck at estimating relative positions using onboard images too,but the mistake is not quite the same as yours.If you successfully run it, can you tell me exactly where the problem is with me?My environment is built entirely according to instructions. error: Traceback (most recent call last): File "", line 1, in runfile('E:/UE4_AirSim_Simulation/Simulation/FormationControl.py', wdir='E:/UE4_AirSim_Simulation/Simulation') File "E:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile execfile(filename, namespace) File "E:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile exec(compile(f.read(), filename, 'exec'), namespace) File "E:/UE4_AirSim_Simulation/Simulation/FormationControl.py", line 180, in Qm, Tm, flagm = meng.GetRelativePose_Ver1_7(imgArray, imgWidth, imgHeight, save, Adjm, itr, nargout=3) File "E:\Anaconda3\lib\site-packages\matlab\engine\matlabengine.py", line 71, in call _stderr, feval=True).result() File "E:\Anaconda3\lib\site-packages\matlab\engine\futureresult.py", line 67, in result return self.__future.result(timeout) File "E:\Anaconda3\lib\site-packages\matlab\engine\fevalfuture.py", line 82, in result self._result = pythonengine.getFEvalResult(self._future,self._nargout, None, out=self._out, err=self._err) MatlabExecutionError: File E:\UE4_AirSim_Simulation\Simulation\QuEst\Helpers\FindTransDepth_Ver1_0.m, line 63, in FindTransDepth_Ver1_0 File E:\UE4_AirSim_Simulation\Simulation\QuEst\QuEst_Ver1_1.m, line 68, in QuEst_Ver1_1 File E:\UE4_AirSim_Simulation\Simulation\QuEst RANSAC\QuEst_RANSAC_Ver1_2.m, line 191, in PoseEstimator File E:\UE4_AirSim_Simulation\Simulation\QuEst RANSAC\QuEst_RANSAC_Ver1_2.m, line 408, in ransac File E:\UE4_AirSim_Simulation\Simulation\QuEst RANSAC\QuEst_RANSAC_Ver1_2.m, line 80, in QuEst_RANSAC_Ver1_2 File E:\UE4_AirSim_Simulation\Simulation\GetRelativePose_Ver1_7.m, line 164, in GetRelativePose_Ver1_7 using svd Input to SVD must not contain NaN or Inf. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>