Regd. VINS-Mono's degradation

[karnikram]

Hi,

I've been running VINS-Mono on your sequences to observe its degradation.

This is the config file I'm using - vins_viode_config.txt. The IMU parameters are the same as yours, and the camera intrinsics are based on a guess since they aren't available yet.

This is the ground truth trajectory for parking_lot environment, extracted from the bagfile - high_groundtruth.txt.

I've run VINS-Mono multiple times on the high-dynamic version of the parking_lot environment multiple times but surprisingly its tracking is still quite stable and the ATE is always less than 0.50 m. This is one sample estimate -
vins_est_traj.txt

But in the paper you've shown VINS-Mono's error to go up to 5 m. Am I missing something? I'm using evo to compute the ATE, with the SE3 alignment option.

evo_ape tum high_groundtruth.txt vins_est_traj.txt --align

[kminoda]

I quickly tried VINS-Mono with your config, and I have also confirmed that the performance is better than the one we reported in our paper.

1. The parameter we used in VINS-Mono (which we focused on in our paper) was not reflected in the VINS-Fusion calibration file in the repository. I've updated the parameter now.
2. It may be because that our parameter tuning on our dataset was not enough. We'll post here if we have any updates about this.

Anyway, thank you very much for the important comment!

[karnikram]

I myself haven't done much tuning, I was using the default VINS-Mono parameters with your camera values, extrinsics, and (old) IMU intrinsic values. I had also disabled relocalization and loop closure.

It's surprising how small changes in the IMU parameters can have a big impact on the accuracy. Thanks for letting me know!

[karnikram]

Update:

I ran VINS-Mono five times again with the updated config file. The error has increased to 0.98 m from earlier. However I feel the increase is more due to the reduced feature count of 120 (from the default 150), because when I run it with the default 150 the error is 0.45 m. Reducing the default feature count might not be the right thing to do.

I also used the true AirSim IMU intrinsics discussed here and converted to discrete time for VINS-Mono based on the reported sampling time of 200Hz.

gyro_noise_density: 0.001234134
gyro_random_walk: 7.0523311e-8
accel_noise_density: 0.033284873
accel_random_walk: 8.8259845e-7

Strangely, like you mentioned earlier, the performance of VINS-Mono is worse here with an error of 2.4 m. Given we are using the right IMU noise values (which are also very less), and that there are no other sources of IMU noise in the simlation, I expected the system to be more robust with these values. But clearly that is not happening and the error is worse than using inflated values. Do you have any insights about this?

Config file:
vins_true_config.txt

[kminoda]

We appreciate all of your insights :)
Currently, we are investigating how these parameters can have an impact on the estimation results.

[karnikram]

As another update, I had noticed that VINS-Mono has a better accuracy with the true Airim IMU parameters in the static sequences than with the inflated parameters. The inflated parameters helps VINS only in the dynamic sequences strangely.

[kminoda]

Thank you for your updates!
We've also run a simple experiment of VINS-Mono on our dataset, changing max_cnt (=max feature number in feature tracking) and IMU noise parameters.
Here's the results.

Those are our observations:

• Using the IMU noise parameters provided by AirSim works pretty bad especially with dynamic sequences (as you've already mentioned)

• Interestingly, increasing max_cnt incredibly improves the estimation of high, while none, low, and mid are in the same trend with what we’ve reported in our paper

• The difference between IMU2 and IMU3 is also somewhat unexpected. high significantly improves while mid significantly degrades.

Still, the overall trend of the estimation error is none < low < mid < high, this is not always true. At least we can say that VINS-Mono is very sensitive to these parameters, especially in dynamic environments (and also to visual features).

Sorry for the late reply. Again, thank you very much for your effort!