The Circular Hough Transform result is often not very accurate due to noise\details\occlusions.
Typical ways of dealing with this are:
1. Hand tuning the Hough Transform parameters.
2. Pre-processing the image aggressively before the transform is applied.
One trick I use to fix the circles positions is an iterative search in windows around the initial circles, I hope to have a future post about this here.
But now I will share a much simpler strategy that works well in some cases: Use CAMSHIFT to track the circular object in a window around the initial circles positions.
The idea is that the initial circle center position area holds information about how the circular object looks like, for example its color distribution. This is complementary to the Hough transform that uses only spatial information (the binary votes in the Hough space).
Find circles with the Circular Hough Transform.
Find the histogram inside a small box around each circle. In a more general case we can use any kind of features we like, like texture features or something, but here we will stick with color features.
For each pixel, find the probability it belongs to the circular object (back-projection).
Optional: apply some strategy to fill holes in the back-projection image caused by occlusions. We can use morphology operations like dilating for example.
Use CAMSHIFT to track the the circular object starting in a window around the initial circle position.
Conveniently for us, CAMSHIFT is included in OpenCV!
I encourage you to read the original CAMSHIFT paper to learn more about it: Computer Vision Face Tracking For Use in a Perceptual User Interface Gary R. Bradski, Microcomputer Research Lab, Santa Clara, CA, Intel Corporation Link to the paper