OpenCV is an incredibly powerful image-processing tool, but it can be difficult to know where to start – how do you grab an image from a camera, and display it in a user-friendly GUI? This post describes such an application, that runs unmodified on a PC or Raspberry Pi, Windows or Linux, Python 2.7 or 3.x, and PyQt v4 or v5.
On Windows, the OpenCV and PyQt5 libraries can be installed using pip:
pip install numpy opencv-python PyQt5
If pip isn’t available, you should be able to run the module from the command line by invoking Python, e.g. for Python 3:
py -3 -m pip install numpy opencv-python PyQt5
Installing on a Raspberry Pi is potentially a lot more complicated; it is generally recommended to install from source, and for opencv-python, this is a bit convoluted. Fortunately there is a simpler option, if you don’t mind using versions that are a few years old, namely to load the binary image from the standard repository, e.g.
sudo apt update sudo apt install python3-opencv python3-pyqt5
At the time of writing, the most recent version of Raspbian Linux is ‘buster’, and that has OpenCV 3.2, which is quite usable. The previous ‘stretch’ distribution has python-opencv version 2.4, which is a bit too old: my code isn’t compatible with it.
With regard to cameras, all the USB Webcams I’ve tried have worked fine on Windows without needing to have any extra driver software installed; they also work on the Raspberry Pi, as well as the standard Pi camera with the ribbon-cable interface.
PyQt main window
Being compatible with PyQt version 4 and 5 requires some boilerplate code to handle the way some functions have been moved between libraries:
import sys, time, threading, cv2 try: from PyQt5.QtCore import Qt pyqt5 = True except: pyqt5 = False if pyqt5: from PyQt5.QtCore import QTimer, QPoint, pyqtSignal from PyQt5.QtWidgets import QApplication, QMainWindow, QTextEdit, QLabel from PyQt5.QtWidgets import QWidget, QAction, QVBoxLayout, QHBoxLayout from PyQt5.QtGui import QFont, QPainter, QImage, QTextCursor else: from PyQt4.QtCore import Qt, pyqtSignal, QTimer, QPoint from PyQt4.QtGui import QApplication, QMainWindow, QTextEdit, QLabel from PyQt4.QtGui import QWidget, QAction, QVBoxLayout, QHBoxLayout from PyQt4.QtGui import QFont, QPainter, QImage, QTextCursor try: import Queue as Queue except: import queue as Queue
The main window is subclassed from PyQt, with a simple arrangement of a menu bar, video image, and text box:
class MyWindow(QMainWindow): text_update = pyqtSignal(str) # Create main window def __init__(self, parent=None): QMainWindow.__init__(self, parent) self.central = QWidget(self) self.textbox = QTextEdit(self.central) self.textbox.setFont(TEXT_FONT) self.textbox.setMinimumSize(300, 100) self.text_update.connect(self.append_text) sys.stdout = self print("Camera number %u" % camera_num) print("Image size %u x %u" % IMG_SIZE) if DISP_SCALE > 1: print("Display scale %u:1" % DISP_SCALE) self.vlayout = QVBoxLayout() # Window layout self.displays = QHBoxLayout() self.disp = ImageWidget(self) self.displays.addWidget(self.disp) self.vlayout.addLayout(self.displays) self.label = QLabel(self) self.vlayout.addWidget(self.label) self.vlayout.addWidget(self.textbox) self.central.setLayout(self.vlayout) self.setCentralWidget(self.central) self.mainMenu = self.menuBar() # Menu bar exitAction = QAction('&Exit', self) exitAction.setShortcut('Ctrl+Q') exitAction.triggered.connect(self.close) self.fileMenu = self.mainMenu.addMenu('&File') self.fileMenu.addAction(exitAction)
There is a horizontal box layout called ‘displays’, that seems to be unnecessary as it only has one display widget in it. This is intentional, since much of my OpenCV experimentation requires additional displays to show the image processing in action; this can easily be done by creating more ImageWidgets, and adding them to the ‘displays’ layout.
Similarly, there is a redundant QLabel below the displays, which isn’t currently used, but is handy for displaying static text below the images.
It is convenient to redirect the ‘print’ output to the text box, rather than appearing on the Python console. This is done using the ‘text_update’ signal that was defined above:
# Handle sys.stdout.write: update text display def write(self, text): self.text_update.emit(str(text)) def flush(self): pass # Append to text display def append_text(self, text): cur = self.textbox.textCursor() # Move cursor to end of text cur.movePosition(QTextCursor.End) s = str(text) while s: head,sep,s = s.partition("\n") # Split line at LF cur.insertText(head) # Insert text at cursor if sep: # New line if LF cur.insertBlock() self.textbox.setTextCursor(cur) # Update visible cursor
The use of a signal means that print() calls can be scattered about the code, without having to worry about which thread they’re in.
A separate thread is used to capture the camera images, and put them in a queue to be displayed. The camera may produce images faster than they can be displayed, so it is necessary to check how many images are already in the queue; if more than 1, the new image is discarded. This prevents a buildup of unwanted images.
IMG_SIZE = 1280,720 # 640,480 or 1280,720 or 1920,1080 CAP_API = cv2.CAP_ANY # or cv2.CAP_DSHOW, etc... EXPOSURE = 0 # Non-zero for fixed exposure # Grab images from the camera (separate thread) def grab_images(cam_num, queue): cap = cv2.VideoCapture(cam_num-1 + CAP_API) cap.set(cv2.CAP_PROP_FRAME_WIDTH, IMG_SIZE) cap.set(cv2.CAP_PROP_FRAME_HEIGHT, IMG_SIZE) if EXPOSURE: cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 0) cap.set(cv2.CAP_PROP_EXPOSURE, EXPOSURE) else: cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1) while capturing: if cap.grab(): retval, image = cap.retrieve(0) if image is not None and queue.qsize() < 2: queue.put(image) else: time.sleep(DISP_MSEC / 1000.0) else: print("Error: can't grab camera image") break cap.release()
The choice of image size will depend on the camera used; all cameras support VGA size (640 x 480 pixels), more modern versions the high-definition standards of 720p (1280 x 720) or 1080p (1920 x 1080).
The camera number refers to the position in the list of cameras collected by the operating system; I’ve defined the first camera as number 1, but the OpenCV call defines the first as 0, so the number has to be adjusted.
The same parameter is also used to define the capture API setting; by default this is ‘any’, which usually works well; my Windows 10 system defaults to the MSMF (Microsoft Media Foundation) backend, while the Raspberry Pi defaults to Video for Linux (V4L). Sometimes you may need to force a particular API to be used, for example, I have a Logitech C270 webcam that works fine on Windows 7, but fails on Windows 10 with an ‘MSMF grab error’. Forcing the software to use the DirectShow API (using the cv2.CAP_DSHOW option) fixes the problem.
If you want to check which backend is being used, try:
print("Backend '%s'" % cap.getBackendName())
Unfortunately this only works on the later revisions of OpenCV.
Manual exposure setting can be a bit hit-and-miss, depending on the camera and API you are using; the default is automatic operation, and setting EXPOSURE non-zero (e.g. to a value of -3) generally works, however it can be difficult to set a webcam back to automatic operation: sometimes I’ve had to use another application to do this. So it is suggested that you keep auto-exposure enabled if possible.
[Supplementary note: it seems that these parameter values aren’t standardised across the backends. For example, the CAP_PROP_AUTO_EXPOSURE value in my source code is correct for the MSMF backend; a value of 1 enables automatic exposure, 0 disables it. However, the V4L backend on the Raspberry Pi uses the opposite values: automatic is 0, and manual is 1. So it looks like my code is incorrect for Linux. I haven’t yet found any detailed documentation for this, so had to fall back on reading the source code, namely the OpenCV videoio ‘cap’ files such as cap_msmf.cpp and cap_v4l.cpp.]
The camera image is displayed in a custom widget:
# Image widget class ImageWidget(QWidget): def __init__(self, parent=None): super(ImageWidget, self).__init__(parent) self.image = None def setImage(self, image): self.image = image self.setMinimumSize(image.size()) self.update() def paintEvent(self, event): qp = QPainter() qp.begin(self) if self.image: qp.drawImage(QPoint(0, 0), self.image) qp.end()
A timer event is used to trigger a scan of the image queue. This contains images in the camera format, which must be converted into the PyQt display format:
DISP_SCALE = 2 # Scaling factor for display image # Start image capture & display def start(self): self.timer = QTimer(self) # Timer to trigger display self.timer.timeout.connect(lambda: self.show_image(image_queue, self.disp, DISP_SCALE)) self.timer.start(DISP_MSEC) self.capture_thread = threading.Thread(target=grab_images, args=(camera_num, image_queue)) self.capture_thread.start() # Thread to grab images # Fetch camera image from queue, and display it def show_image(self, imageq, display, scale): if not imageq.empty(): image = imageq.get() if image is not None and len(image) > 0: img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) self.display_image(img, display, scale) # Display an image, reduce size if required def display_image(self, img, display, scale=1): disp_size = img.shape//scale, img.shape//scale disp_bpl = disp_size * 3 if scale > 1: img = cv2.resize(img, disp_size, interpolation=cv2.INTER_CUBIC) qimg = QImage(img.data, disp_size, disp_size, disp_bpl, IMG_FORMAT) display.setImage(qimg)
This demonstrates the power of OpenCV; with one function call we convert the image from BGR to RGB format, then another is used to resize the image using cubic interpolation. Finally a PyQt function is used to convert from OpenCV to PyQt format.
Running the application
Make sure you’re using the Python version that has the OpenCV and PyQt installed, e.g. for the Raspberry Pi:
There is an optional argument that can be used if there are multiple cameras; the default first camera is number 1.
On Linux, some USB Webcams cause a constant stream of JPEG format errors to be printed on the console, complaining about extraneous bytes in the data. There is some discussion online as to the cause of the error, and the cure seems to involve rebuilding the libraries from source; I’m keen to avoid that, so used the simple workaround of suppressing the errors by redirecting STDERR to null:
python3 cam_display.py 2> /dev/null
Fortunately this workaround is only needed with some USB cameras; the standard Raspberry Pi camera with the CSI ribbon-cable interface works fine.
Full source code is available here.
For a more significant OpenCV application, take a look at this post.
Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.