Remove Moving Objects from Video in OpenCV Python using Background Subtraction

Introduction

Ever wondered how they remove moving objects from the video frames while giving special effects in the movies. A lot goes into giving this effect. You can however create similar effects to remove moving objects from video in OpenCV Python by using background subtraction techniques. In this article, we will show you to implement this by using the two methods – i) Running Average and ii) Median Filter with a couple of examples.

What is Background Subtraction?

Background subtraction is the process of separating the background and foreground from a sequence of image/video frames. It is generally used for detecting or removing moving objects from the videos of static cameras. The subtraction method should:

  1. Take into account spatial scales of objects and should adapt to sudden and gradual changes.
  2. Classify pixels as background or foreground and segment them.
  3. Adapt to geometry, motion, illumination.
Remove moving objects from video in OpenCV Python using Background Subtraction
Background Subtraction

Moving Object Removal from Videos with OpenCV Python

Installing the required libraries

We will start by installing OpenCV Python and Numpy libraries if they are not already installed in your system. Run the below commands in the terminal.

pip install opencv-python
pip install numpy

Importing Required Libraries

From here on out we will start to write some code, so open up a python file in your favorite code editor and import our required libraries as shown below.

import cv2
import numpy as np

Sample Videos for Example

We will be applying our background extraction algorithms in OpenCV Python to remove moving objects from the video shown below. Here the moving objects are vehicles on the road.

Example Video 1
Example Video 1
Example video 2
Example video 2

Technique 1 – Running Average

The concept of running average is to detect active objects and remove them i.e. differentiate between pixels that seem to change over time and remove them. Here “Running” signifies the fact that the average is being computed over previous and current frames again and again until the frames are exhausted. Simply, the background is the mean of ‘n’ previous frames in the video.

Running average formula
Running average formula

Python Implementation

file_path = 'video.mp4'
cap = cv2.VideoCapture(file_path)
first_iter = True
result1 = None
while True:
    ret, frame = cap.read()
    if frame is None:
        break

    if first_iter:
       avg = np.float32(frame)
       first_iter = False

   cv2.accumulateWeighted(frame, avg, 0.005)
   result1 = cv2.convertScaleAbs(avg)

 

Line 1-2: Initialize a video object using the ‘cv2.VideoCapture’ method.

Line 3-4: We define some variables, 1) is the ‘first_iter'(with a ‘True’ value) it will help us fetch the first frame of the video later (so stay tuned!). 2) ‘result1’ will hold our final result so we initialize it beforehand.

Line 5: Initialize a while loop that runs indefinitely or until it has looped through all the frames of the video(line 5) and the next frame is ‘None’. (line 7-8)

Line 10-11: For the first iteration of the loop the value of ‘first_iter’ is true thus the code inside the ‘if’ statement on line 9 will be executed. The first frame of the video will be converted to a Numpy array and saved in a variable called ‘avg’ and the ‘first_iter’ will be set to false.

Line 14: We call the ‘cv2.accumulateWeighted’ function to computes the average of consecutive frames with every iteration. The first argument is the image of which the average is to be computed. The second argument is the destination image to which the computed average is to be applied. Third, the ‘alpha’ decides the speed of updating the weight of the input image. (Example: A lower value results in the running average being performed over a larger amount of previous frames and vice-versa)

Line 15: As the name suggests this function(‘cv2.convertScaleAbs’) computes the absolute for each value of the pixel. (The exact process is scaling, calculating absolute, and converting to uint8 type)

The value of ‘result’ will be updated with each iteration of the loop until all the frames are looped through.

Results

Add the following code to your python file to show the output of the code.

cv2.imshow("Running average result", result1) cv2.waitKey(0)

 

cv2.imshow("Running average result", result1)
cv2.waitKey(0)
Remove moving objects from video in OpenCV Python using Background Subtraction - Running Average
Running Average applied
Remove moving objects from video in OpenCV Python using Background Subtraction - Running Average
Running Average applied

Technique 2 – Median filtering

Temporal median filtering is one of a kind background subtraction method that approximates the median of each pixel to be the background. We can make the assumption that the median pixel is background due to the fact that the background is most likely to be in the video the longest. Thus we can apply this logic to frames as well and say that the background frame is equal to the mean of the previous ‘n’ frames. However, median filtering is a computationally heavy and time-consuming process thus its applications are limited.

Median filtering formula
Median filtering formula

Python Implementation

file_path = 'video.mp4'
video = cv2.VideoCapture(file_path)

FOI = video.get(cv2.CAP_PROP_FRAME_COUNT) * np.random.uniform(size=30)

frames = []
for frameOI in FOI:
    video.set(cv2.CAP_PROP_POS_FRAMES, frameOI)
    ret, frame = video.read()
    frames.append(frame)

result2 = np.median(frames, axis=0).astype(dtype=np.uint8)

 

Line 1-2: Define the path to the video and load it to memory using ‘cv.VideoCapture’.

Line 4: Later we will be applying the median functions on the frames of our video. Unfortunately, that process converts our frame into a 1-dimensional array instead of maintaining three(width, height, channels) dimensions. Thus we need to augment our frames with random values and in this step, we achieve that by multiplying the no of frames in the video(cv2.CAP_PROP_POS_FRAMES fetches the no. of frames) by an array of size (30,1) filled with random values.

Line 6: Define an empty array(‘frames’) that will hold the video frames that were created in the previous step after we loop through them.

Line 7-8: Initialize a ‘for’ loop to loop through the modified frames that were created on line 4. Inside the for loop, we access our original video’s current frame(cv2.CAP_PROP_POS_FRAMES refers to the 0 based index of the frame to be decoded/captured next) property using the ‘set()’ function (which is called on a ‘VideoCapture’ object) and change it to the corresponding modified frame (i.e. ‘frameOI’). Then fetch the frame and append it to our ‘frames’ array.

Line 12: Call the numpy ‘median’ function to calculate the median across the frames array. Since we will be computing the median it will be like the function is returning a single image(because of the ‘uint8’ type) that lies exactly in the middle of the array. (pixel-wise middle)

Results

In order to see the computed background image add the following code to the end of the code.

cv2.imshow("Median filtering result",result2) 
cv2.waitKey(0)

 

Remove moving objects from video in OpenCV Python using Background Subtraction - Median Filter
Median Filter applied
Remove moving objects from video in OpenCV Python using Background Subtraction - Median Filter
Median Filter applied

Conclusion

Our tutorial showed how we can use OpenCV Python to remove moving objects in video using background subtraction. But as you may see the results are not very good always with these techniques. There are breakthroughs in deep learning that is helping researchers to achieve object removal with far more accuracy than ever before. That will be a topic for a different article, however.

  • Gaurav Maindola

    I am a machine learning enthusiast with a keen interest in web development. My main interest is in the field of computer vision and I am fascinated with all things that comprise making computers learn and love to learn new things myself.

    View all posts

Follow Us

Leave a Reply

Your email address will not be published. Required fields are marked *