OpenCV face & facial landmark tracker class derived from CVTracked. More...

#include <CVTrackedFaces.h>

Inheritance diagram for CVTrackedFaces:

Public Member Functions
	CVTrackedFaces (string faceClassifierFilename, string faceMarkModelFilename, int smoothLength=5)
	Constructor for the facial landmark tracker. More...

	~CVTrackedFaces ()

bool	track (CVMat imageGray, CVMat imageBgr, CVCalibration *calib) final
	Tracks the a face and its landmarks. More...

Public Member Functions inherited from CVTracked
	CVTracked ()

virtual	~CVTracked ()=default

void	drawDetection (bool draw)

bool	isVisible ()

bool	drawDetection ()

CVMatx44f	objectViewMat ()

Static Public Member Functions
static void	delaunayTriangulate (CVMat imageBgr, const CVVPoint2f &points, bool drawDetection)

Static Public Member Functions inherited from CVTracked
static cv::Matx44f	createGLMatrix (const CVMat &tVec, const CVMat &rVec)
	Create an OpenGL 4x4 matrix from an OpenCV translation & rotation vector. More...

static void	createRvecTvec (const CVMatx44f &glMat, CVMat &tVec, CVMat &rVec)
	Creates the OpenCV rvec & tvec vectors from an column major OpenGL 4x4 matrix. More...

static CVMatx44f	calcObjectMatrix (const CVMatx44f &cameraObjectMat, const CVMatx44f &objectViewMat)

static CVVec3f	averageVector (vector< CVVec3f > vectors, vector< float > weights)

static SLQuat4f	averageQuaternion (vector< SLQuat4f > quaternions, vector< float > weights)

static void	resetTimes ()
	Resets all static variables. More...

Private Attributes
CVCascadeClassifier *	_faceDetector
	Viola-Jones face detector. More...

cv::Ptr< CVFacemark >	_facemark
	Facial landmarks detector smart pointer. More...

vector< AvgCVVec2f >	_avgPosePoints2D
	vector of averaged facial landmark 2D points More...

CVRect	_boundingRect
	Bounding rectangle around landmarks. More...

CVVPoint2f	_cvPosePoints2D
	vector of OpenCV point2D More...

CVVPoint3f	_cvPosePoints3D
	vector of OpenCV point2D More...

int	_smoothLength
	Smoothing filter lenght. More...

Additional Inherited Members
Static Public Attributes inherited from CVTracked
static AvgFloat	trackingTimesMS
	Averaged time for video tracking in ms. More...

static AvgFloat	detectTimesMS
	Averaged time for video feature detection & description in ms. More...

static AvgFloat	detect1TimesMS
	Averaged time for video feature detection subpart 1 in ms. More...

static AvgFloat	detect2TimesMS
	Averaged time for video feature detection subpart 2 in ms. More...

static AvgFloat	matchTimesMS
	Averaged time for video feature matching in ms. More...

static AvgFloat	optFlowTimesMS
	Averaged time for video feature optical flow tracking in ms. More...

static AvgFloat	poseTimesMS
	Averaged time for video feature pose estimation in ms. More...

Protected Attributes inherited from CVTracked
bool	_isVisible
	Flag if marker is visible. More...

bool	_drawDetection
	Flag if detection should be drawn into image. More...

CVMatx44f	_objectViewMat
	view transformation matrix More...

HighResTimer	_timer
	High resolution timer. More...

Detailed Description

OpenCV face & facial landmark tracker class derived from CVTracked.

Tracking class for face and facial landmark tracking. The class uses the OpenCV face detection algorithm from Viola-Jones to find all faces in the image and the facial landmark detector provided in cv::facemark. For more details see the comments in CVTrackedFaces::track method.

Definition at line 34 of file CVTrackedFaces.h.

Constructor & Destructor Documentation

◆ CVTrackedFaces()

CVTrackedFaces::CVTrackedFaces	(	string	faceClassifierFilename,
		string	faceMarkModelFilename,
		int	smoothLength = `5`
	)

explicit

Constructor for the facial landmark tracker.

The Constructor loads the training files for the face and the facial landmarks. It also defines an averaged set of facial landmark points in 3D used for pose estimation.

Parameters

faceClassifierFilename	Name of the cascaded face training file
faceMarkModelFilename	Name of the facial landmark training file
smoothLength	Length of averaging filter

Definition at line 31 of file CVTrackedFaces.cpp.

 {
     // Load Haar cascade training file for the face detection
     if (!Utils::fileExists(faceClassifierFilename))
     {
         string msg = "CVTrackedFaces: File not found: " + faceClassifierFilename;
         Utils::exitMsg("SLProject", msg.c_str(), __LINE__, __FILE__);
     }
     _faceDetector = new CVCascadeClassifier(faceClassifierFilename);
  
     // Load facemark model file for the facial landmark detection
     if (!Utils::fileExists(faceMarkModelFilename))
     {
         string msg = "CVTrackedFaces: File not found: " + faceMarkModelFilename;
         Utils::exitMsg("SLProject", msg.c_str(), __LINE__, __FILE__);
     }
  
     cv::face::FacemarkLBF::Params facemarkParams;
     facemarkParams.verbose = false; // no logging
     _facemark              = cv::face::FacemarkLBF::create(facemarkParams);
     _facemark->loadModel(faceMarkModelFilename);
  
     // Init averaged 2D facial landmark points
     _smoothLength = smoothLength;
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Nose tip
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Nose hole left
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Nose hole right
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Left eye left corner
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Left eye right corner
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Right eye left corner
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Right eye right corner
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Left mouth corner
     _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Right mouth corner
  
     _cvPosePoints2D.resize(_avgPosePoints2D.size(), CVPoint2f(0, 0));
  
     // Set 3D facial points in mm
     _cvPosePoints3D.push_back(CVPoint3f(.000f, .000f, .000f));    // Nose tip
     _cvPosePoints3D.push_back(CVPoint3f(-.015f, -.005f, -.018f)); // Nose hole left
     _cvPosePoints3D.push_back(CVPoint3f(.015f, -.005f, -.018f));  // Nose hole right
     _cvPosePoints3D.push_back(CVPoint3f(-.047f, .041f, -.036f));  // Left eye left corner
     _cvPosePoints3D.push_back(CVPoint3f(-.019f, .041f, -.033f));  // Left eye right corner
     _cvPosePoints3D.push_back(CVPoint3f(.019f, .041f, -.033f));   // Right eye left corner
     _cvPosePoints3D.push_back(CVPoint3f(.047f, .041f, -.036f));   // Right eye right corner
     _cvPosePoints3D.push_back(CVPoint3f(-.025f, -.035f, -.036f)); // Left Mouth corner
     _cvPosePoints3D.push_back(CVPoint3f(.025f, -.035f, -.036f));  // Right mouth corner
 }

◆ ~CVTrackedFaces()

CVTrackedFaces::~CVTrackedFaces ( )

Definition at line 81 of file CVTrackedFaces.cpp.

 {
     delete _faceDetector;
 }

Member Function Documentation

◆ delaunayTriangulate()

void CVTrackedFaces::delaunayTriangulate	(	CVMat	imageBgr,
		const CVVPoint2f &	points,
		bool	drawDetection
	)

static

Returns the Delaunay triangulation on the points within the image

Parameters

imageBgr	OpenCV BGR image
points	2D points as OpenCV vector of points 2D
drawDetection	Flag if detection should be drawn

Definition at line 241 of file CVTrackedFaces.cpp.

 {
     // Get rect of image
     CVSize size = imageBgr.size();
     CVRect rect(0, 0, size.width, size.height);
  
     // Create an instance of Subdiv2D
     cv::Subdiv2D subdiv(rect);
  
     // Do Delaunay triangulation for the landmarks
     for (const CVPoint2f& point : points)
         if (rect.contains(point))
             subdiv.insert(point);
  
     // Add additional points in the corners and middle of the sides
     float w = (float)size.width;
     float h = (float)size.height;
     subdiv.insert(CVPoint2f(0.0f, 0.0f));
     subdiv.insert(CVPoint2f(w * 0.5f, 0.0f));
     subdiv.insert(CVPoint2f(w - 1.0f, 0.0f));
     subdiv.insert(CVPoint2f(w - 1.0f, h * 0.5f));
     subdiv.insert(CVPoint2f(w - 1.0f, h - 1.0f));
     subdiv.insert(CVPoint2f(w * 0.5f, h - 1.0f));
     subdiv.insert(CVPoint2f(0.0f, h - 1.0f));
     subdiv.insert(CVPoint2f(0.0f, h * 0.5f));
  
     // Draw Delaunay triangles
     if (drawDetection)
     {
         vector<cv::Vec6f> triangleList;
         subdiv.getTriangleList(triangleList);
         CVVPoint pt(3);
  
         for (auto t : triangleList)
         {
             pt[0] = CVPoint(cvRound(t[0]), cvRound(t[1]));
             pt[1] = CVPoint(cvRound(t[2]), cvRound(t[3]));
             pt[2] = CVPoint(cvRound(t[4]), cvRound(t[5]));
  
             // Draw rectangles completely inside the image.
             if (rect.contains(pt[0]) && rect.contains(pt[1]) && rect.contains(pt[2]))
             {
                 line(imageBgr, pt[0], pt[1], cv::Scalar(255, 255, 255), 1, cv::LINE_AA, 0);
                 line(imageBgr, pt[1], pt[2], cv::Scalar(255, 255, 255), 1, cv::LINE_AA, 0);
                 line(imageBgr, pt[2], pt[0], cv::Scalar(255, 255, 255), 1, cv::LINE_AA, 0);
             }
         }
     }
 }

◆ track()

bool CVTrackedFaces::track	(	CVMat	imageGray,
		CVMat	imageBgr,
		CVCalibration *	calib
	)

finalvirtual

Tracks the a face and its landmarks.

Implements CVTracked.

Definition at line 100 of file CVTrackedFaces.cpp.

 {
     PROFILE_FUNCTION();
  
     assert(!imageGray.empty() && "ImageGray is empty");
     assert(!imageBgr.empty() && "ImageBGR is empty");
     assert(!calib->cameraMat().empty() && "Calibration is empty");
  
     //////////////////
     // Detect Faces //
     //////////////////
  
     float startMS = _timer.elapsedTimeInMilliSec();
  
     // Detect faces
     CVVRect faces;
     int     min = (int)((float)imageGray.rows * 0.4f); // the bigger min the faster
     int     max = (int)((float)imageGray.rows * 0.8f); // the smaller max the faster
     CVSize  minSize(min, min);
     CVSize  maxSize(max, max);
     _faceDetector->detectMultiScale(imageGray,
                                     faces,
                                     1.05,
                                     3,
                                     0,
                                     minSize,
                                     maxSize);
  
     // Enlarge the face rect at the bottom to cover also the chin
     for (auto& face : faces)
         face.height = (int)(face.height * 1.2f);
  
     float time2MS = _timer.elapsedTimeInMilliSec();
     CVTracked::detect1TimesMS.set(time2MS - startMS);
  
     //////////////////////
     // Detect Landmarks //
     //////////////////////
  
     CVVVPoint2f lm;
     bool        foundLandmarks = _facemark->fit(imageBgr, faces, lm);
  
     float time3MS = _timer.elapsedTimeInMilliSec();
     CVTracked::detect2TimesMS.set(time3MS - time2MS);
     CVTracked::detectTimesMS.set(time3MS - startMS);
  
     if (foundLandmarks)
     {
         for (unsigned long i = 0; i < lm.size(); i++)
         {
             // Landmark indexes from
             // https://cdn-images-1.medium.com/max/1600/1*AbEg31EgkbXSQehuNJBlWg.png
             _avgPosePoints2D[0].set(CVVec2f(lm[i][30].x, lm[i][30].y)); // Nose tip
             _avgPosePoints2D[1].set(CVVec2f(lm[i][31].x, lm[i][31].y)); // Nose hole left
             _avgPosePoints2D[2].set(CVVec2f(lm[i][35].x, lm[i][35].y)); // Nose hole right
             _avgPosePoints2D[3].set(CVVec2f(lm[i][36].x, lm[i][36].y)); // Left eye left corner
             _avgPosePoints2D[4].set(CVVec2f(lm[i][39].x, lm[i][39].y)); // Left eye right corner
             _avgPosePoints2D[5].set(CVVec2f(lm[i][42].x, lm[i][42].y)); // Right eye left corner
             _avgPosePoints2D[6].set(CVVec2f(lm[i][45].x, lm[i][45].y)); // Right eye right corner
             _avgPosePoints2D[7].set(CVVec2f(lm[i][48].x, lm[i][48].y)); // Left mouth corner
             _avgPosePoints2D[8].set(CVVec2f(lm[i][54].x, lm[i][54].y)); // Right mouth corner
  
             // Convert averaged 2D points to OpenCV points2d
             for (unsigned long p = 0; p < _avgPosePoints2D.size(); p++)
                 _cvPosePoints2D[p] = CVPoint2f(_avgPosePoints2D[p].average()[0],
                                                _avgPosePoints2D[p].average()[1]);
  
             // delaunayTriangulate(imageBgr, landmarks[i], drawDetection);
  
             ///////////////////
             // Visualization //
             ///////////////////
  
             if (_drawDetection)
             {
                 // Draw rectangle of detected face
                 cv::rectangle(imageBgr,
                               faces[i],
                               cv::Scalar(255, 0, 0),
                               2);
  
                 // Draw detected landmarks
                 for (auto& j : lm[i])
                     cv::circle(imageBgr,
                                j,
                                2,
                                cv::Scalar(0, 0, 255),
                                -1);
  
                 // Draw averaged face points used for pose estimation
                 for (unsigned long p = 0; p < _avgPosePoints2D.size(); p++)
                     cv::circle(imageBgr,
                                _cvPosePoints2D[p],
                                5,
                                cv::Scalar(0, 255, 0),
                                1);
             }
  
             // Do pose estimation for the first face found
             if (i == 0)
             {
                 /////////////////////
                 // Pose Estimation //
                 /////////////////////
  
                 startMS = _timer.elapsedTimeInMilliSec();
  
                 // find the camera extrinsic parameters (rVec & tVec)
                 CVMat rVec; // rotation angle vector as axis (length as angle)
                 CVMat tVec; // translation vector
                 bool  solved = solvePnP(CVMat(_cvPosePoints3D),
                                        CVMat(_cvPosePoints2D),
                                        calib->cameraMat(),
                                        calib->distortion(),
                                        rVec,
                                        tVec,
                                        false,
                                        cv::SOLVEPNP_EPNP);
  
                 CVTracked::poseTimesMS.set(_timer.elapsedTimeInMilliSec() - startMS);
  
                 if (solved)
                 {
                     _objectViewMat = createGLMatrix(tVec, rVec);
                     return true;
                 }
             }
         }
     }
  
     return false;
 }

Member Data Documentation

◆ _avgPosePoints2D

vector<AvgCVVec2f> CVTrackedFaces::_avgPosePoints2D

private

vector of averaged facial landmark 2D points

Definition at line 53 of file CVTrackedFaces.h.

◆ _boundingRect

CVRect CVTrackedFaces::_boundingRect

private

Bounding rectangle around landmarks.

Definition at line 54 of file CVTrackedFaces.h.

◆ _cvPosePoints2D

CVVPoint2f CVTrackedFaces::_cvPosePoints2D

private

vector of OpenCV point2D

Definition at line 55 of file CVTrackedFaces.h.

◆ _cvPosePoints3D

CVVPoint3f CVTrackedFaces::_cvPosePoints3D

private

vector of OpenCV point2D

Definition at line 56 of file CVTrackedFaces.h.

◆ _faceDetector

CVCascadeClassifier* CVTrackedFaces::_faceDetector

private

Viola-Jones face detector.

Definition at line 51 of file CVTrackedFaces.h.

◆ _facemark

cv::Ptr<CVFacemark> CVTrackedFaces::_facemark

private

Facial landmarks detector smart pointer.

Definition at line 52 of file CVTrackedFaces.h.

◆ _smoothLength

int CVTrackedFaces::_smoothLength

private

Smoothing filter lenght.

Definition at line 57 of file CVTrackedFaces.h.

The documentation for this class was generated from the following files:

Public Member Functions

Static Public Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ CVTrackedFaces()

◆ ~CVTrackedFaces()

Member Function Documentation

◆ delaunayTriangulate()

◆ track()

Member Data Documentation

◆ _avgPosePoints2D

◆ _boundingRect

◆ _cvPosePoints2D

◆ _cvPosePoints3D

◆ _faceDetector

◆ _facemark

◆ _smoothLength