Chances are, you already at least heard the term augmented reality. The very generic definition says that augmented reality is a real-world enhanced by computer-generated information. This mini-series will be about enhancing the captured image or video by adding a 3D model into it. Like in this video.
If you want to know how this works, then keep reading. I will walk you through all the steps. The following image is the rough summarization of the steps.

Now in simple words. The final goal is to display a cube on top of the white A4 paper. The process to do that starts by thresholding the pixels to find the white ones. This produces several blobs. Assuming the A4 paper is the biggest of them. Others are suppressed. Next step is to identify contour, edge lines and corners of the paper. Then the planar transformation is computed. Planar transformation is used to compute a projection matrix which allows drawing 3D objects to the original image. Finally, the projection matrix is transformed into the OpenGL compatible form. Then you can do anything with that.
Sounds trivial, right? Maybe. Still, it takes some effort to go through the details, therefore I have prepared the following chapters.
In addition, there is an example project accompanying this series. You can download it right here.
Code is written in Java (+ few OpenGL shaders) and build by Maven. As soon as you understand these, you should be able to build the project and run the test applications. Test applications are executable classes within test sources. Main classes are CameraPoseVideoTestApp and CameraPoseJoglTestApp.
Some extra reading? I used these resources (valid as of August 2019).
- https://bitesofcode.wordpress.com/2017/09/12/augmented-reality-with-python-and-opencv-part-1/
- http://ksimek.github.io/2012/08/13/introduction/
- http://www.uco.es/investiga/grupos/ava/node/26
- http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf
- https://en.wikipedia.org/wiki/Camera_resectioning
- https://docs.opencv.org/3.4.3/dc/dbb/tutorial_py_calibration.html
- Gordon, V. Scott. Computer Graphics Programming in OpenGL with JAVA Second Edition
- Solem, Jan Erik. Programming Computer Vision with Python
Summary
This chapter gave you an overall idea of the project. Next chapter will tell you how to track the plane.