In this repository I have combined three functionalities to detect pose of target objects held by hand in dark. Please watch my demo video for more details.
Following are the repositories/ideas I have relied on:
- I have used DOPE (Deep Object Pose Estimation) for tracking object pose.
- YOLO2 PyTorch impelementation by marvis (pytorch-yolo2) for tracking hands wearing gloves.
- Gamma correction using OpenCV (Link) for processing dark/low-exposure frames.
My objective in this project is to detect a medical tool called a cautery in surgical lighting. A cautery is a electronic knife like tool that looks like a pen. It is used to make cuts during surgery and is a very extensively used tool.
As part of my research on Augmented Surgical Reality, I investigate the usability of existing 6DoF object pose estimation methods such as Deep Object Pose Estimation (DOPE) and Single Shot Pose (SSP) for cautery tool detection. After applying DOPE and SSP, I realized that the cautery is a very difficult object to track for multiple reasons. Most important reasons are: the cautery is very small and thin, featureless, is bound to be heavily occluded. One other problem that is unique to Operation Room environment only, is the high-intensity surgical lighting above. To handle this I tried lowering the exposure of the camera. However, due to such high intensity of the lighting the exposure has to be lowered to such a low level that neither YOLO2 nor DOPE is able to detect anything (DOPE in extreme lighting demo video). This all makes it very challenging to track a cautery pen in operation room like environment.
In order to overcome these problems in this project, I tried out merging 3 different methods. To rectify the dark image (due to low exposure settings of camera) I added a gamma correction methods which makes the image brighter. Following are two captured images before and after the gamma correction respectively. Notice how the image gets brighter after gamma correction. A demonstration of the gamma correction and how it helps tracking hands in the dark is demonstrated in this video.
I trained YOLO2 to track hands. Following is a demo image of hand tracking using YOLO2. I trained my own weights for hand tracking instead of using the existing hand-trackers. That is because existing hand-trackers track only bare hands without gloves. If there are gloves, then the trackers fail. My tracker can work with different colored hands and will different gloves.
Finally, I take a cropped square image around the hand which is then run
through DOPE to identify the pose of the pen. The operation speed of
DOPE increases according to the size of selected crop size around the
detected hand. DOPE with hand tracking looks like the following image in
normal lighting. The green box is centered at the detected hand and the
portion inside the green box is being cropped. The yellow 3D bounding
box shows the detected pose of the target object.
After my experiments, I realized that I lacked proper dataset to train DOPE for being able to detect objects in the dark. I demostrate that using the following image where I try to detect the well known ycb-object 'spam can'. I used the original weights provided by the authors of DOPE but still DOPE was not able to detect the can even with the help of hand detection. This is because the lighting condition used by the authors are not as extreme as mine( Please take a look at the Falling Objects dataset). Currently I am working on making better dataset.
I uploaded the necessary weights in my Dropbox.
Put the 'backup' folder in the project direcory.
Please follow instructions for DOPE.
I have included the requirements.txt file from the original DOPE repo. Here is a demo for installation.
The original pytorch-yolo2 was workable only with PyTorch version 0.3.1. I have changed the codes so that it can work with 0.4.0 which DOPE uses.
I have used an Intel Realsense D435i camera for this implementation as I needed to control exposure.
live_dope_hand_realsense.py is the final implementation. In the code change the "Settings" section if needed. Please watch my demo video for more details.
live_dope_hand_webcam.py is implementation with webcam. In the code change the "Settings" section if needed. The only difference between this one and the realsense one is that, the webcam doesn't have exposure control. Here is a demo video.
live_dope_realsense.py and live_dope_webcam.py are just DOPE demos using the realsense camera and webcam respectively. The original DOPE demos are using ROS. I needed a simple demo so I made these two.
Please comment/uncomment weights in my_config_realsense.yaml and my_config_webcam.yaml files for selecting which object you want to detect.
One problem I was facing was that my computer was not detecting webcam while it was being connected to the realsense camera. After wasting a lot of time, I realized the solution is to just connect one camera at a time. Weird problem.