来自屏幕截图的模板匹配答案

【问题标题】：Template matching from screen capture来自屏幕截图的模板匹配
【发布时间】：2019-10-06 22:40:57
【问题描述】：

我是 Python 新手，但想学一点，所以我决定创建一个程序与来自桌面输入的模板匹配。

有人可以帮忙吗？如何编写与桌面流匹配的模板？

import time

import cv2
import mss
import numpy

template = cv2.imread('template.jpg', 0)
w, h = template.shape[::-1]

with mss.mss() as sct:
    # Part of the screen to capture
    monitor = {"top": 40, "left": 0, "width": 800, "height": 640}

    while "Screen capturing":
        last_time = time.time()

        # Get raw pixels from the screen, save it to a Numpy array
        img = numpy.array(sct.grab(monitor))

        # Display the picture
        # cv2.imshow("OpenCV/Numpy normal", img)

        # Display the picture in grayscale
        cv2.imshow('OpenCV/Numpy grayscale', cv2.cvtColor(img, cv2.COLOR_BGRA2GRAY))

        # Print fps
        print("fps: {}".format(1 / (time.time() - last_time)))

        # Search template in stream

        # Press "q" to quit
        if cv2.waitKey(25) & 0xFF == ord("q"):
            cv2.destroyAllWindows()
            break

【问题讨论】：

标签： python opencv

【解决方案1】：

以下是执行匹配模板的基本代码。将其放在img = numpy.array(sct.grab(monitor)) 下方，它将运行每一帧。

# create grayscale of image - because template is also grayscale
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# perform match
res = cv2.matchTemplate(gray,template ,cv2.TM_CCOEFF)
# get coordinates of best match
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
top_left = min_loc
bottom_right = (top_left[0] + w, top_left[1] + h)
# draw red rectangle over original screen capture
cv2.rectangle(img,top_left, bottom_right,(0,0,255),3)
# display image
cv2.imshow('Result',img)

您可以在 matchTemplate here 上找到更多信息

【讨论】：

【解决方案2】：

设untitled.png为存储图片的文件

这是一个工作程序。我用下面的把它放在一起，

OpenCV Org: Template Matching

Taking screenshots with OpenCV and Python

OpenCV python: ValueError: too many values to unpack

import os
import cv2               as cv
import numpy             as np
import pyautogui
import time
import winsound          # for sound
from matplotlib          import pyplot as plt

os.chdir("C:\\Users\\Mike\\\Desktop")
img = cv.imread('untitled.png',0)
img_piece = cv.cvtColor(img, cv.COLOR_RGB2BGR)
c, w, h = img_piece.shape[::-1]

while 1:
    pic = pyautogui.screenshot()
    template  = cv.cvtColor(np.array(pic), cv.COLOR_RGB2BGR)
    meth = 'cv.TM_CCOEFF'
    method = eval(meth)
    # Apply template Matching
    res = cv.matchTemplate(img_piece,template,method)
    min_val, max_val, min_loc, max_loc = cv.minMaxLoc(res)
    top_left = max_loc
    bottom_right = (top_left[0] + w, top_left[1] + h)
    cv.rectangle(img,top_left, bottom_right, 255, 2)
    if max_val > 66000000.0:
        print(max_val, top_left, bottom_right)
        winsound.Beep(888, 111)
        if 1:
            plt.subplot(121),plt.imshow(res,cmap = 'gray')
            plt.title('Matching Result'), plt.xticks([]), plt.yticks([])
            plt.subplot(122),plt.imshow(img,cmap = 'gray')
            plt.title('Detected Point'), plt.xticks([]), plt.yticks([])
            plt.suptitle(meth)
            plt.show()        
        break
    time.sleep(1)

【讨论】：

【解决方案3】：

我注意到的第一件事是您没有对模板图像应用任何边缘检测。边缘检测不是必需的，但对于查找模板图像的特征很有用。

假设我有以下图片：

为了准确检测上述模板图像，我应该应用边缘检测算法。

template = cv2.imread("three.png")
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
template = cv2.Canny(template, 50, 200)

我还应该对来自桌面的流应用边缘检测。

img = sct.grab(mon)
gray = cv2.cvtColor(np.array(img), cv2.COLOR_BGR2GRAY)
edged = cv2.Canny(gray, 50, 200)

检查模板是否与捕获的图像匹配

result = cv2.matchTemplate(edged, template, cv2.TM_CCOEFF)
(_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

如果模板图像在来自桌面的流中匹配，则获取坐标。

(_, maxLoc, r) = found
(startX, startY) = (int(maxLoc[0] * r), int(maxLoc[1] * r))
(endX, endY) = (int((maxLoc[0] + w) * r), int((maxLoc[1] + h) * r))

最后画出显示位置的矩形：

cv2.rectangle(img, (startX, startY), (endX, endY), (180, 105, 255), 2)

结果：

从上面我们看到我们的模板 3 值与来自桌面的流匹配。

代码：

import time
import cv2
import numpy as np
import imutils
from mss import mss

template = cv2.imread("three.png")
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
template = cv2.Canny(template, 50, 200)
(h, w) = template.shape[:2]

start_time = time.time()
mon = {'top': 200, 'left': 200, 'width': 200, 'height': 200}
with mss() as sct:
    while True:
        last_time = time.time()
        img = sct.grab(mon)
        img = np.array(img)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        edged = cv2.Canny(gray, 50, 200)
        
        found = None

        for scale in np.linspace(0.2, 1.0, 20)[::-1]:
            resized = imutils.resize(gray, width=int(gray.shape[1] * scale))
            r = gray.shape[1] / float(resized.shape[1])

            if resized.shape[0] < h or resized.shape[1] < w:
                break

            edged = cv2.Canny(resized, 50, 200)
            cv2.imwrite("canny_image.png", edged)
            result = cv2.matchTemplate(edged, template, cv2.TM_CCOEFF)
            (_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

            if found is None or maxVal > found[0]:
                found = (maxVal, maxLoc, r)

        (_, maxLoc, r) = found
        (startX, startY) = (int(maxLoc[0] * r), int(maxLoc[1] * r))
        (endX, endY) = (int((maxLoc[0] + w) * r), int((maxLoc[1] + h) * r))

        cv2.rectangle(img, (startX, startY), (endX, endY), (180, 105, 255), 2)

        print('The loop took: {0}'.format(time.time()-last_time))
        cv2.imshow('test', np.array(img))

        if cv2.waitKey(25) & 0xFF == ord('q'):
            cv2.destroyAllWindows()
            break

【讨论】：