[FIXED] How to automatically find a rectangle and crop in OpenCV or PIL

Issue

I have a program that uses PIL to automatically crop a series of images to a certain area. However, when I run the program with screenshots from a different sized display, the area cropped is in the wrong place. Is there a way to use OpenCV or PIL to automatically find the rectangle that I want to crop to (for example the main viewer of a Youtube video) and crop to it, while leaving the image in color and then saving the image to a new folder?

My code for cropping images:

import os, random
from PIL import Image

files = []

for x in os.listdir():
    if '.jpg' in f'{x}' or '.png' in f'{x}' or '.jpeg' in f'{x}':
        files.append(x)
    else:
        print(f'Passed by {x}! It is not an image!')
for x in files:
    y = hex(random.randint(100000,500000))
    image = Image.open(f'{x}')
    newimage = image.crop((48,367,1626,1256))
    newimage.save(f'newdir/{y}.png')

The example image (this works with the PIL cropper):

The image I want:

Another image from another computer that needs to be cropped to the same viewer:

Solution

Here is one way to do that using Python/OpenCV.

Basically, threshold the image, then get contours, then get the bounding box of the largest contour and crop using the bounding box.

Input:

import cv2
import numpy as np

# load image
img = cv2.imread("screen.jpg")

# get color bounds of white region
lower =(180,180,180) # lower bound for each channel
upper = (255,255,255) # upper bound for each channel

# threshold
threshold = cv2.inRange(img, lower, upper)

# get the largest contour
contours = cv2.findContours(threshold, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = contours[0] if len(contours) == 2 else contours[1]
big_contour = max(contours, key=cv2.contourArea)

# get bounding box
x,y,w,h = cv2.boundingRect(big_contour)
print(x,y,w,h)


# crop the image at the bounds
crop = img[y:y+h, x:x+w]

# write result to disk
cv2.imwrite("screen_threshold.jpg", threshold)
cv2.imwrite("screen_cropped.jpg", crop)

# display it
cv2.imshow("threshold", threshold)
cv2.imshow("crop", crop)
cv2.waitKey(0)

Threshold Image:

Cropped Result:

Cropped (x,y,w,h):

48 368 1578 801

Answered By - fmw42

Answer Checked By - Katrina (PHPFixing Volunteer)