python - How can I get a list of objects that appeared newly in an image with OpenCv?

Question

I'm trying to identify a list of objects that appeared newly in a photo. The plan is to get multiple cropped images from the original image and feed them to a neural network for object detection. Right now, I'm having trouble in extracting objects that appeared in a frame.

import cv2 as cv
import matplotlib.pyplot as plt

def mdisp(image):
    plt.imshow(image)
    plt.show()

im1 = cv.imread('images/litter-before.jpg')
mdisp(im1)
print(im1.shape)
im2 = cv.imread('images/litter-after.jpg')
mdisp(im2)
print(im2.shape)
backsub1=cv.createBackgroundSubtractorMOG2()
backsub2=cv.createBackgroundSubtractorKNN()
fgmask = backsub1.apply(im1)
fgmask = backsub1.apply(im2)
print(fgmask.shape)
mdisp(fgmask)
new_image = im2 * (fgmask[:,:,None].astype(im2.dtype))
mdisp(new_image)

Ideally, I would like to get a cropped picture of the item within red circle. How can I do it with OpenCv

Answer 1

Here's an approach, subtracting the two frames directly. The idea is that you first convert your images to grayscale, then blur a little bit to ignore the noise. Subtract the two frames, threshold the difference and look for the largest blob that is above a certain area threshold value.

Let's see:

import cv2
import numpy as np

# image path
path = "C:/opencvImages/"
fileName01 = "01.jpg"
fileName02 = "02.jpg"

# Read the2 images in default mode:
image01 = cv2.imread(path + fileName01)
image02 = cv2.imread(path + fileName02)

# Store a copy of the last frame for results drawing:
inputCopy = image02.copy()

# Convert RGB images to grayscale:
grayscaleImage01 = cv2.cvtColor(image01, cv2.COLOR_BGR2GRAY)
grayscaleImage02 = cv2.cvtColor(image02, cv2.COLOR_BGR2GRAY)

# Convert RGB images to grayscale:
filterSize = 5
imageMedian01 = cv2.medianBlur(grayscaleImage01, filterSize)
imageMedian02 = cv2.medianBlur(grayscaleImage02, filterSize)

Now you have the grayscale, blurred frames. Next, we need to calculate the difference between these frames. I don't wanna loose data, so I have to be careful with the data type here. Remember that these are grayscale, uint8 matrices, but the difference could potentially yield negative values. Let's convert the matrices to floats, take the difference, and convert this matrix to uint8:

# uint8 to float32 conversion:
imageMedian01 = imageMedian01.astype('float32')
imageMedian02 = imageMedian02.astype('float32')

# Take the difference and convert back to uint8
imageDifference = np.clip(imageMedian01 - imageMedian02, 0, 255)
imageDifference = imageDifference.astype('uint8')

This gives you the frames difference:

Let's threshold this to get a binary image. I'm using a threshold value of 127, as it is a the center of the 8-bit range:

threshValue = 127
_, binaryImage = cv2.threshold(imageDifference, threshValue, 255, cv2.THRESH_BINARY)

This is the binary image:

We are looking for the biggest blob here, let's find blob/contours and filter the small ones. Let's set a minimum area of 10 pixels:

# Perform an area filter on the binary blobs:
componentsNumber, labeledImage, componentStats, componentCentroids = 
    cv2.connectedComponentsWithStats(binaryImage, connectivity=4)

# Set the minimum pixels for the area filter:
minArea = 10

# Get the indices/labels of the remaining components based on the area stat
# (skip the background component at index 0)
remainingComponentLabels = [i for i in range(1, componentsNumber) if componentStats[i][4] >= minArea]

# Filter the labeled pixels based on the remaining labels,
# assign pixel intensity to 255 (uint8) for the remaining pixels
filteredImage = np.where(np.isin(labeledImage, remainingComponentLabels) == True, 255, 0).astype('uint8')

# Find the big contours/blobs on the filtered image:
contours, hierarchy = cv2.findContours(filteredImage, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)

contours_poly = [None] * len(contours)
boundRect = []

# Alright, just look for the outer bounding boxes:
for i, c in enumerate(contours):
    if hierarchy[0][i][3] == -1:
        contours_poly[i] = cv2.approxPolyDP(c, 3, True)
        boundRect.append(cv2.boundingRect(contours_poly[i]))


# Draw the bounding boxes on the (copied) input image:
for i in range(len(boundRect)):
    print(boundRect[i])
    color = (0, 255, 0)
    cv2.rectangle(inputCopy, (int(boundRect[i][0]), int(boundRect[i][1])), 
                  (int(boundRect[i][0] + boundRect[i][2]), int(boundRect[i][1] + boundRect[i][3])), color, 1)

Check out the results: