Project

Developing a multithreaded real time video processing application in OpenCV


See how real time video processing is done using OpenCV. We go by an example and discuss the implementation of an algorithm to detect breathing rate of a person in video. Witness how to make an application multi-threaded to provide a smooth user experience.



Introduction


The challenge is straight forward, from a video, detect breathing rate of a person, who is either sleeping or is relatively still. In this previous post we discussed an algorithm to find the frequency of breath inhale and exhales. The idea is to analyse the pixel changes at a location in the video related to the body movements induced because of respiration. By applying Fourier Transformation to the time series of pixel changes, we get the breathing rate. I have implemented this algorithm in C++, which is available here. In this post, I discuss the major features of the implementation, which I find could also be useful for any real time video processing application.



Outline


Lets see what we want to have by the end of this project. There should be an app which should take a video as argument, could be either web-cam live feed or a video file. Then, the UI would request the user to select a rectangular region in which the processing would be restricted, this would increase speed and reduce noise. Finally, we expect the video being played with the breathing rate shown somewhere inside, all in real time and as smooth as possible.



Code Features


  • The app should continuously fetch fresh frames from the video feed and display it on a GUI window

  • Since we need to process a sequence of frames in real time, we sample a short length video, say of 2 seconds, and process that to evaluate the breathing rate

  • The processing may take some reasonable amount of time, which could reduce the display frame-rate. To avoid this, we do the processing and UI on separate threads

  • The processing thread needs a sample (sequence of frames) to process and evaluate. Hence, there has to be an array of frames which is shared between the processing thread and the frames fetching thread

  • The producer thread (frames fetching thread), is producing faster than it could be consumed by the processing thread. Hence the frames are fetched in a large buffer circular array, of which the processing thread takes a small part as sample

  • A variable is shared between the processing thread and the UI thread, which contains the current breathing rate. This variable is updated by the processing thread, whenever it evaluates the breathing rate on a sample, while the UI thread keeps displaying the value on the UI window

  • All the threads keep looping infinitely, the producer thread fetching frames, the processing threads evaluating breathing rate, and the UI thread displaying the video feed and evaluated breathing rate




Producer Thread


In each step of its loop, the thread reads a frame from the video input and puts it into the frame buffer, which is a cyclic array. This is usually a fast procedure, the bottleneck being the speed at which the video feed generates a frame.

void RealTimeVideo::putFrameInBuffer(Mat &f){

pos = idx % buffLen; // Cyclic Array

frameBuffer[pos] = f.clone();

idx++;

}

void RealTimeVideo::producer(){

idx = 0; // Initialise frame counter

Mat f;

while(1){

capture->read(f);

frame = f.clone(); // Update the current frame for UI

putFrameInBuffer(f);

}

}



UI Thread


This thread is responsible for displaying the output to the user. The output in our case is the original video feed with the calculated breathing rate imprinted on it. Illusion of video is achieved by showing each captured frame on a window in a very small interval of time. Before displaying, the output result in a shared variable is imprinted on the frame using an OpenCV function.

void RealTimeVideo::showFrameOutput(){

drawText(frame, to_string(breathingRate)); // Draw the processed result to the output

imshow("output", frame);

}

void RealTimeVideo::UI(){

while(true){

waitKey(20); // Frame display delay

showFrameOutput();

}

}



Processing Thread


This thread contains the master logic of our video processing, where all the heavy, time consuming processing is carried out. It first retrieves a small chunk of the frame buffer, which it processes, generating a result, which is breathing rate in our case. This result is updated in a variable shared with the UI thread, for that thread to display it as the output.

vector<Mat> RealTimeVideo::getSample(){

int end = pos ; // pos indicates the latest frame index in the frame buffer

int st = pos - sampleLen; // Extract a sample of size sampleLen

if(st < 0)

st = 0;

vector<Mat> ret(frameBuffer.begin()+st, frameBuffer.begin()+end); // Extract a slice from buffer

return ret ;

}

float RealTimeVideo::processSample(vector<Mat> sample){

float result = APPLY_FFT(sample) ; // Perform all the heavy calculations here

return result;

}

void RealTimeVideo::processor(){

vector<Mat> sample;

while(1){

sample = getSample();

breathingRate = processSample(sample); // Updating shared result variable

}

}



Processing and Fourier Transformation


This method evaluates the breathing rate from the time series of pixel variations of the video. Applying Fourier Transformation gives us the magnitudes of a range of frequencies in the input signal. We filter out the improbable signals and find the pixel and frequency which is highest in magnitude. This frequency corresponds to the actual breathing rate.

float APPLY_FFT(Mat &timeSeries, float low_frequency, float high_frequency, int samplingRate){

cv::Mat fft;

int n = timeSeries.cols ;

cv::dft(timeSeries, fft, DFT_ROWS); // Perform Discrete Fourier Transform over every row

int low_ind = (low_frequency * n ) / samplingRate ;

int high_ind = ( high_frequency * n ) / samplingRate ; // Band pass filtering

float amp ,x,y, curMax = 0, freqMax;

for(int j=0; j < timeSeries.rows; j++){

for(int i=low_ind-1; i< high_ind+1; i++){

x = fft.at<double>(j , 2*i - 1);

y = fft.at<double>(j , 2*i); // Thats how OpenCV outputs the frequency spectrum

amp = x*x + y*y ; // Amplitude of the corresponding frequency

if(amp > curMax){

curMax = amp;

freqMax = (float)(i* samplingRate)/(float)(n) ;

}

}

}

return freqMax*60 ;// Convert to breadths per minute

}



Launching all the threads


To execute each thread, we use the C++ thread library as shown in the code below. The main thread waits for the producer thread to finish before the whole program exits.

void RealTimeVideo::runThreads(){

thread producer_t(&RealTimeVideo::producer, this);

thread UI_t(&RealTimeVideo::UI, this);

thread processor_t(&RealTimeVideo::processor, this);

producer_t.join();

}



Conclusion


That was the core architecture of a real time video processing application. We divided the work into separate threads, video input output being on one set of threads, while the computationally intensive part being on a separate thread. These threads communicated with each other using shared variables providing a smooth interface.

Do have a look at the class structure of RealTimeVideo in my implementation. You can clone the repository and compile it on your machine to play around with it. For that you will require OpenCV and CMake installed on your system.

Rohan Raja

Recently graduated, majoring in Mathematics and Computing from IIT Kharagpur, Rohan is a technology enthusiast and passionate programmer. Likes to apply Mathematics and Artificial Intelligence to devise creative solutions to common problems.