Doxygen tutorials: python basic

doc/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.markdown

@@ -0,0 +1,173 @@
+Background Subtraction {#tutorial_py_bg_subtraction}
+======================
+
+Goal
+----
+
+In this chapter,
+
+-   We will familiarize with the background subtraction methods available in OpenCV.
+
+Basics
+------
+
+Background subtraction is a major preprocessing steps in many vision based applications. For
+example, consider the cases like visitor counter where a static camera takes the number of visitors
+entering or leaving the room, or a traffic camera extracting information about the vehicles etc. In
+all these cases, first you need to extract the person or vehicles alone. Technically, you need to
+extract the moving foreground from static background.
+
+If you have an image of background alone, like image of the room without visitors, image of the road
+without vehicles etc, it is an easy job. Just subtract the new image from the background. You get
+the foreground objects alone. But in most of the cases, you may not have such an image, so we need
+to extract the background from whatever images we have. It become more complicated when there is
+shadow of the vehicles. Since shadow is also moving, simple subtraction will mark that also as
+foreground. It complicates things.
+
+Several algorithms were introduced for this purpose. OpenCV has implemented three such algorithms
+which is very easy to use. We will see them one-by-one.
+
+### BackgroundSubtractorMOG
+
+It is a Gaussian Mixture-based Background/Foreground Segmentation Algorithm. It was introduced in
+the paper "An improved adaptive background mixture model for real-time tracking with shadow
+detection" by P. KadewTraKuPong and R. Bowden in 2001. It uses a method to model each background
+pixel by a mixture of K Gaussian distributions (K = 3 to 5). The weights of the mixture represent
+the time proportions that those colours stay in the scene. The probable background colours are the
+ones which stay longer and more static.
+
+While coding, we need to create a background object using the function,
+**cv2.createBackgroundSubtractorMOG()**. It has some optional parameters like length of history,
+number of gaussian mixtures, threshold etc. It is all set to some default values. Then inside the
+video loop, use backgroundsubtractor.apply() method to get the foreground mask.
+
+See a simple example below:
+@code{.py}
+import numpy as np
+import cv2
+
+cap = cv2.VideoCapture('vtest.avi')
+
+fgbg = cv2.createBackgroundSubtractorMOG()
+
+while(1):
+    ret, frame = cap.read()
+
+    fgmask = fgbg.apply(frame)
+
+    cv2.imshow('frame',fgmask)
+    k = cv2.waitKey(30) & 0xff
+    if k == 27:
+        break
+
+cap.release()
+cv2.destroyAllWindows()
+@endcode
+( All the results are shown at the end for comparison).
+
+### BackgroundSubtractorMOG2
+
+It is also a Gaussian Mixture-based Background/Foreground Segmentation Algorithm. It is based on two
+papers by Z.Zivkovic, "Improved adaptive Gausian mixture model for background subtraction" in 2004
+and "Efficient Adaptive Density Estimation per Image Pixel for the Task of Background Subtraction"
+in 2006. One important feature of this algorithm is that it selects the appropriate number of
+gaussian distribution for each pixel. (Remember, in last case, we took a K gaussian distributions
+throughout the algorithm). It provides better adaptibility to varying scenes due illumination
+changes etc.
+
+As in previous case, we have to create a background subtractor object. Here, you have an option of
+selecting whether shadow to be detected or not. If detectShadows = True (which is so by default), it
+detects and marks shadows, but decreases the speed. Shadows will be marked in gray color.
+@code{.py}
+import numpy as np
+import cv2
+
+cap = cv2.VideoCapture('vtest.avi')
+
+fgbg = cv2.createBackgroundSubtractorMOG2()
+
+while(1):
+    ret, frame = cap.read()
+
+    fgmask = fgbg.apply(frame)
+
+    cv2.imshow('frame',fgmask)
+    k = cv2.waitKey(30) & 0xff
+    if k == 27:
+        break
+
+cap.release()
+cv2.destroyAllWindows()
+@endcode
+(Results given at the end)
+
+### BackgroundSubtractorGMG
+
+This algorithm combines statistical background image estimation and per-pixel Bayesian segmentation.
+It was introduced by Andrew B. Godbehere, Akihiro Matsukawa, Ken Goldberg in their paper "Visual
+Tracking of Human Visitors under Variable-Lighting Conditions for a Responsive Audio Art
+Installation" in 2012. As per the paper, the system ran a successful interactive audio art
+installation called “Are We There Yet?” from March 31 - July 31 2011 at the Contemporary Jewish
+Museum in San Francisco, California.
+
+It uses first few (120 by default) frames for background modelling. It employs probabilistic
+foreground segmentation algorithm that identifies possible foreground objects using Bayesian
+inference. The estimates are adaptive; newer observations are more heavily weighted than old
+observations to accommodate variable illumination. Several morphological filtering operations like
+closing and opening are done to remove unwanted noise. You will get a black window during first few
+frames.
+
+It would be better to apply morphological opening to the result to remove the noises.
+@code{.py}
+import numpy as np
+import cv2
+
+cap = cv2.VideoCapture('vtest.avi')
+
+kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(3,3))
+fgbg = cv2.createBackgroundSubtractorGMG()
+
+while(1):
+    ret, frame = cap.read()
+
+    fgmask = fgbg.apply(frame)
+    fgmask = cv2.morphologyEx(fgmask, cv2.MORPH_OPEN, kernel)
+
+    cv2.imshow('frame',fgmask)
+    k = cv2.waitKey(30) & 0xff
+    if k == 27:
+        break
+
+cap.release()
+cv2.destroyAllWindows()
+@endcode
+Results
+-------
+
+**Original Frame**
+
+Below image shows the 200th frame of a video
+
+![image](images/resframe.jpg)
+
+**Result of BackgroundSubtractorMOG**
+
+![image](images/resmog.jpg)
+
+**Result of BackgroundSubtractorMOG2**
+
+Gray color region shows shadow region.
+
+![image](images/resmog2.jpg)
+
+**Result of BackgroundSubtractorGMG**
+
+Noise is removed with morphological opening.
+
+![image](images/resgmg.jpg)
+
+Additional Resources
+--------------------
+
+Exercises
+---------

doc/py_tutorials/py_video/py_lucas_kanade/py_lucas_kanade.markdown

@@ -0,0 +1,226 @@
+Optical Flow {#tutorial_py_lucas_kanade}
+============
+
+Goal
+----
+
+In this chapter,
+   -   We will understand the concepts of optical flow and its estimation using Lucas-Kanade
+        method.
+    -   We will use functions like **cv2.calcOpticalFlowPyrLK()** to track feature points in a
+        video.
+
+Optical Flow
+------------
+
+Optical flow is the pattern of apparent motion of image objects between two consecutive frames
+caused by the movemement of object or camera. It is 2D vector field where each vector is a
+displacement vector showing the movement of points from first frame to second. Consider the image
+below (Image Courtesy: [Wikipedia article on Optical
+Flow](http://en.wikipedia.org/wiki/Optical_flow)).
+
+![image](images/optical_flow_basic1.jpg)
+
+It shows a ball moving in 5 consecutive frames. The arrow shows its displacement vector. Optical
+flow has many applications in areas like :
+
+-   Structure from Motion
+-   Video Compression
+-   Video Stabilization ...
+
+Optical flow works on several assumptions:
+
+-#  The pixel intensities of an object do not change between consecutive frames.
+2.  Neighbouring pixels have similar motion.
+
+Consider a pixel \f$I(x,y,t)\f$ in first frame (Check a new dimension, time, is added here. Earlier we
+were working with images only, so no need of time). It moves by distance \f$(dx,dy)\f$ in next frame
+taken after \f$dt\f$ time. So since those pixels are the same and intensity does not change, we can say,
+
+\f[I(x,y,t) = I(x+dx, y+dy, t+dt)\f]
+
+Then take taylor series approximation of right-hand side, remove common terms and divide by \f$dt\f$ to
+get the following equation:
+
+\f[f_x u + f_y v + f_t = 0 \;\f]
+
+where:
+
+\f[f_x = \frac{\partial f}{\partial x} \; ; \; f_y = \frac{\partial f}{\partial x}\f]\f[u = \frac{dx}{dt} \; ; \; v = \frac{dy}{dt}\f]
+
+Above equation is called Optical Flow equation. In it, we can find \f$f_x\f$ and \f$f_y\f$, they are image
+gradients. Similarly \f$f_t\f$ is the gradient along time. But \f$(u,v)\f$ is unknown. We cannot solve this
+one equation with two unknown variables. So several methods are provided to solve this problem and
+one of them is Lucas-Kanade.
+
+### Lucas-Kanade method
+
+We have seen an assumption before, that all the neighbouring pixels will have similar motion.
+Lucas-Kanade method takes a 3x3 patch around the point. So all the 9 points have the same motion. We
+can find \f$(f_x, f_y, f_t)\f$ for these 9 points. So now our problem becomes solving 9 equations with
+two unknown variables which is over-determined. A better solution is obtained with least square fit
+method. Below is the final solution which is two equation-two unknown problem and solve to get the
+solution.
+
+\f[\begin{bmatrix} u \\ v \end{bmatrix} =
+\begin{bmatrix}
+    \sum_{i}{f_{x_i}}^2  &  \sum_{i}{f_{x_i} f_{y_i} } \\
+    \sum_{i}{f_{x_i} f_{y_i}} & \sum_{i}{f_{y_i}}^2
+\end{bmatrix}^{-1}
+\begin{bmatrix}
+    - \sum_{i}{f_{x_i} f_{t_i}} \\
+    - \sum_{i}{f_{y_i} f_{t_i}}
+\end{bmatrix}\f]
+
+( Check similarity of inverse matrix with Harris corner detector. It denotes that corners are better
+points to be tracked.)
+
+So from user point of view, idea is simple, we give some points to track, we receive the optical
+flow vectors of those points. But again there are some problems. Until now, we were dealing with
+small motions. So it fails when there is large motion. So again we go for pyramids. When we go up in
+the pyramid, small motions are removed and large motions becomes small motions. So applying
+Lucas-Kanade there, we get optical flow along with the scale.
+
+Lucas-Kanade Optical Flow in OpenCV
+-----------------------------------
+
+OpenCV provides all these in a single function, **cv2.calcOpticalFlowPyrLK()**. Here, we create a
+simple application which tracks some points in a video. To decide the points, we use
+**cv2.goodFeaturesToTrack()**. We take the first frame, detect some Shi-Tomasi corner points in it,
+then we iteratively track those points using Lucas-Kanade optical flow. For the function
+**cv2.calcOpticalFlowPyrLK()** we pass the previous frame, previous points and next frame. It
+returns next points along with some status numbers which has a value of 1 if next point is found,
+else zero. We iteratively pass these next points as previous points in next step. See the code
+below:
+@code{.py}
+import numpy as np
+import cv2
+
+cap = cv2.VideoCapture('slow.flv')
+
+# params for ShiTomasi corner detection
+feature_params = dict( maxCorners = 100,
+                       qualityLevel = 0.3,
+                       minDistance = 7,
+                       blockSize = 7 )
+
+# Parameters for lucas kanade optical flow
+lk_params = dict( winSize  = (15,15),
+                  maxLevel = 2,
+                  criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))
+
+# Create some random colors
+color = np.random.randint(0,255,(100,3))
+
+# Take first frame and find corners in it
+ret, old_frame = cap.read()
+old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
+p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params)
+
+# Create a mask image for drawing purposes
+mask = np.zeros_like(old_frame)
+
+while(1):
+    ret,frame = cap.read()
+    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+
+    # calculate optical flow
+    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
+
+    # Select good points
+    good_new = p1[st==1]
+    good_old = p0[st==1]
+
+    # draw the tracks
+    for i,(new,old) in enumerate(zip(good_new,good_old)):
+        a,b = new.ravel()
+        c,d = old.ravel()
+        mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
+        frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
+    img = cv2.add(frame,mask)
+
+    cv2.imshow('frame',img)
+    k = cv2.waitKey(30) & 0xff
+    if k == 27:
+        break
+
+    # Now update the previous frame and previous points
+    old_gray = frame_gray.copy()
+    p0 = good_new.reshape(-1,1,2)
+
+cv2.destroyAllWindows()
+cap.release()
+@endcode
+(This code doesn't check how correct are the next keypoints. So even if any feature point disappears
+in image, there is a chance that optical flow finds the next point which may look close to it. So
+actually for a robust tracking, corner points should be detected in particular intervals. OpenCV
+samples comes up with such a sample which finds the feature points at every 5 frames. It also run a
+backward-check of the optical flow points got to select only good ones. Check
+samples/python2/lk_track.py).
+
+See the results we got:
+
+![image](images/opticalflow_lk.jpg)
+
+Dense Optical Flow in OpenCV
+----------------------------
+
+Lucas-Kanade method computes optical flow for a sparse feature set (in our example, corners detected
+using Shi-Tomasi algorithm). OpenCV provides another algorithm to find the dense optical flow. It
+computes the optical flow for all the points in the frame. It is based on Gunner Farneback's
+algorithm which is explained in "Two-Frame Motion Estimation Based on Polynomial Expansion" by
+Gunner Farneback in 2003.
+
+Below sample shows how to find the dense optical flow using above algorithm. We get a 2-channel
+array with optical flow vectors, \f$(u,v)\f$. We find their magnitude and direction. We color code the
+result for better visualization. Direction corresponds to Hue value of the image. Magnitude
+corresponds to Value plane. See the code below:
+@code{.py}
+import cv2
+import numpy as np
+cap = cv2.VideoCapture("vtest.avi")
+
+ret, frame1 = cap.read()
+prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)
+hsv = np.zeros_like(frame1)
+hsv[...,1] = 255
+
+while(1):
+    ret, frame2 = cap.read()
+    next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)
+
+    flow = cv2.calcOpticalFlowFarneback(prvs,next, None, 0.5, 3, 15, 3, 5, 1.2, 0)
+
+    mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1])
+    hsv[...,0] = ang*180/np.pi/2
+    hsv[...,2] = cv2.normalize(mag,None,0,255,cv2.NORM_MINMAX)
+    rgb = cv2.cvtColor(hsv,cv2.COLOR_HSV2BGR)
+
+    cv2.imshow('frame2',rgb)
+    k = cv2.waitKey(30) & 0xff
+    if k == 27:
+        break
+    elif k == ord('s'):
+        cv2.imwrite('opticalfb.png',frame2)
+        cv2.imwrite('opticalhsv.png',rgb)
+    prvs = next
+
+cap.release()
+cv2.destroyAllWindows()
+@endcode
+See the result below:
+
+![image](images/opticalfb.jpg)
+
+OpenCV comes with a more advanced sample on dense optical flow, please see
+samples/python2/opt_flow.py.
+
+Additional Resources
+--------------------
+
+Exercises
+---------
+
+-#  Check the code in samples/python2/lk_track.py. Try to understand the code.
+2.  Check the code in samples/python2/opt_flow.py. Try to understand the code.
+

doc/py_tutorials/py_video/py_meanshift/py_meanshift.markdown

@@ -0,0 +1,186 @@
+Meanshift and Camshift {#tutorial_py_meanshift}
+======================
+
+Goal
+----
+
+In this chapter,
+
+-   We will learn about Meanshift and Camshift algorithms to find and track objects in videos.
+
+Meanshift
+---------
+
+The intuition behind the meanshift is simple. Consider you have a set of points. (It can be a pixel
+distribution like histogram backprojection). You are given a small window ( may be a circle) and you
+have to move that window to the area of maximum pixel density (or maximum number of points). It is
+illustrated in the simple image given below:
+
+![image](images/meanshift_basics.jpg)
+
+The initial window is shown in blue circle with the name "C1". Its original center is marked in blue
+rectangle, named "C1_o". But if you find the centroid of the points inside that window, you will
+get the point "C1_r" (marked in small blue circle) which is the real centroid of window. Surely
+they don't match. So move your window such that circle of the new window matches with previous
+centroid. Again find the new centroid. Most probably, it won't match. So move it again, and continue
+the iterations such that center of window and its centroid falls on the same location (or with a
+small desired error). So finally what you obtain is a window with maximum pixel distribution. It is
+marked with green circle, named "C2". As you can see in image, it has maximum number of points. The
+whole process is demonstrated on a static image below:
+
+![image](images/meanshift_face.gif)
+
+So we normally pass the histogram backprojected image and initial target location. When the object
+moves, obviously the movement is reflected in histogram backprojected image. As a result, meanshift
+algorithm moves our window to the new location with maximum density.
+
+### Meanshift in OpenCV
+
+To use meanshift in OpenCV, first we need to setup the target, find its histogram so that we can
+backproject the target on each frame for calculation of meanshift. We also need to provide initial
+location of window. For histogram, only Hue is considered here. Also, to avoid false values due to
+low light, low light values are discarded using **cv2.inRange()** function.
+@code{.py}
+import numpy as np
+import cv2
+
+cap = cv2.VideoCapture('slow.flv')
+
+# take first frame of the video
+ret,frame = cap.read()
+
+# setup initial location of window
+r,h,c,w = 250,90,400,125  # simply hardcoded the values
+track_window = (c,r,w,h)
+
+# set up the ROI for tracking
+roi = frame[r:r+h, c:c+w]
+hsv_roi =  cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
+mask = cv2.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))
+roi_hist = cv2.calcHist([hsv_roi],[0],mask,[180],[0,180])
+cv2.normalize(roi_hist,roi_hist,0,255,cv2.NORM_MINMAX)
+
+# Setup the termination criteria, either 10 iteration or move by atleast 1 pt
+term_crit = ( cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1 )
+
+while(1):
+    ret ,frame = cap.read()
+
+    if ret == True:
+        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
+        dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1)
+
+        # apply meanshift to get the new location
+        ret, track_window = cv2.meanShift(dst, track_window, term_crit)
+
+        # Draw it on image
+        x,y,w,h = track_window
+        img2 = cv2.rectangle(frame, (x,y), (x+w,y+h), 255,2)
+        cv2.imshow('img2',img2)
+
+        k = cv2.waitKey(60) & 0xff
+        if k == 27:
+            break
+        else:
+            cv2.imwrite(chr(k)+".jpg",img2)
+
+    else:
+        break
+
+cv2.destroyAllWindows()
+cap.release()
+@endcode
+Three frames in a video I used is given below:
+
+![image](images/meanshift_result.jpg)
+
+Camshift
+--------
+
+Did you closely watch the last result? There is a problem. Our window always has the same size when
+car is farther away and it is very close to camera. That is not good. We need to adapt the window
+size with size and rotation of the target. Once again, the solution came from "OpenCV Labs" and it
+is called CAMshift (Continuously Adaptive Meanshift) published by Gary Bradsky in his paper
+"Computer Vision Face Tracking for Use in a Perceptual User Interface" in 1988.
+
+It applies meanshift first. Once meanshift converges, it updates the size of the window as,
+\f$s = 2 \times \sqrt{\frac{M_{00}}{256}}\f$. It also calculates the orientation of best fitting ellipse
+to it. Again it applies the meanshift with new scaled search window and previous window location.
+The process is continued until required accuracy is met.
+
+![image](images/camshift_face.gif)
+
+### Camshift in OpenCV
+
+It is almost same as meanshift, but it returns a rotated rectangle (that is our result) and box
+parameters (used to be passed as search window in next iteration). See the code below:
+@code{.py}
+import numpy as np
+import cv2
+
+cap = cv2.VideoCapture('slow.flv')
+
+# take first frame of the video
+ret,frame = cap.read()
+
+# setup initial location of window
+r,h,c,w = 250,90,400,125  # simply hardcoded the values
+track_window = (c,r,w,h)
+
+# set up the ROI for tracking
+roi = frame[r:r+h, c:c+w]
+hsv_roi =  cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
+mask = cv2.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))
+roi_hist = cv2.calcHist([hsv_roi],[0],mask,[180],[0,180])
+cv2.normalize(roi_hist,roi_hist,0,255,cv2.NORM_MINMAX)
+
+# Setup the termination criteria, either 10 iteration or move by atleast 1 pt
+term_crit = ( cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1 )
+
+while(1):
+    ret ,frame = cap.read()
+
+    if ret == True:
+        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
+        dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1)
+
+        # apply meanshift to get the new location
+        ret, track_window = cv2.CamShift(dst, track_window, term_crit)
+
+        # Draw it on image
+        pts = cv2.boxPoints(ret)
+        pts = np.int0(pts)
+        img2 = cv2.polylines(frame,[pts],True, 255,2)
+        cv2.imshow('img2',img2)
+
+        k = cv2.waitKey(60) & 0xff
+        if k == 27:
+            break
+        else:
+            cv2.imwrite(chr(k)+".jpg",img2)
+
+    else:
+        break
+
+cv2.destroyAllWindows()
+cap.release()
+@endcode
+Three frames of the result is shown below:
+
+![image](images/camshift_result.jpg)
+
+Additional Resources
+--------------------
+
+-#  French Wikipedia page on [Camshift](http://fr.wikipedia.org/wiki/Camshift). (The two animations
+    are taken from here)
+2.  Bradski, G.R., "Real time face and object tracking as a component of a perceptual user
+    interface," Applications of Computer Vision, 1998. WACV '98. Proceedings., Fourth IEEE Workshop
+    on , vol., no., pp.214,219, 19-21 Oct 1998
+
+Exercises
+---------
+
+-#  OpenCV comes with a Python sample on interactive demo of camshift. Use it, hack it, understand
+    it.
+

doc/py_tutorials/py_video/py_table_of_contents_video/py_table_of_contents_video.markdown

@@ -0,0 +1,16 @@
+Video Analysis {#tutorial_py_table_of_contents_video}
+==============
+
+-   @subpage tutorial_py_meanshift
+
+    We have already seen
+    an example of color-based tracking. It is simpler. This time, we see significantly better
+    algorithms like "Meanshift", and its upgraded version, "Camshift" to find and track them.
+
+-   @subpage tutorial_py_lucas_kanade
+
+    Now let's discuss an important concept, "Optical Flow", which is related to videos and has many applications.
+
+-   @subpage tutorial_py_bg_subtraction
+
+    In several applications, we need to extract foreground for further operations like object tracking. Background Subtraction is a well-known method in those cases.