SLProject  4.2.000
A platform independent 3D computer graphics framework for desktop OS, Android, iOS and online in web browsers
CVTrackedFaces Class Reference

OpenCV face & facial landmark tracker class derived from CVTracked. More...

#include <CVTrackedFaces.h>

Inheritance diagram for CVTrackedFaces:
[legend]

Public Member Functions

 CVTrackedFaces (string faceClassifierFilename, string faceMarkModelFilename, int smoothLength=5)
 Constructor for the facial landmark tracker. More...
 
 ~CVTrackedFaces ()
 
bool track (CVMat imageGray, CVMat imageBgr, CVCalibration *calib) final
 Tracks the a face and its landmarks. More...
 
- Public Member Functions inherited from CVTracked
 CVTracked ()
 
virtual ~CVTracked ()=default
 
void drawDetection (bool draw)
 
bool isVisible ()
 
bool drawDetection ()
 
CVMatx44f objectViewMat ()
 

Static Public Member Functions

static void delaunayTriangulate (CVMat imageBgr, const CVVPoint2f &points, bool drawDetection)
 
- Static Public Member Functions inherited from CVTracked
static cv::Matx44f createGLMatrix (const CVMat &tVec, const CVMat &rVec)
 Create an OpenGL 4x4 matrix from an OpenCV translation & rotation vector. More...
 
static void createRvecTvec (const CVMatx44f &glMat, CVMat &tVec, CVMat &rVec)
 Creates the OpenCV rvec & tvec vectors from an column major OpenGL 4x4 matrix. More...
 
static CVMatx44f calcObjectMatrix (const CVMatx44f &cameraObjectMat, const CVMatx44f &objectViewMat)
 
static CVVec3f averageVector (vector< CVVec3f > vectors, vector< float > weights)
 
static SLQuat4f averageQuaternion (vector< SLQuat4f > quaternions, vector< float > weights)
 
static void resetTimes ()
 Resets all static variables. More...
 

Private Attributes

CVCascadeClassifier_faceDetector
 Viola-Jones face detector. More...
 
cv::Ptr< CVFacemark_facemark
 Facial landmarks detector smart pointer. More...
 
vector< AvgCVVec2f_avgPosePoints2D
 vector of averaged facial landmark 2D points More...
 
CVRect _boundingRect
 Bounding rectangle around landmarks. More...
 
CVVPoint2f _cvPosePoints2D
 vector of OpenCV point2D More...
 
CVVPoint3f _cvPosePoints3D
 vector of OpenCV point2D More...
 
int _smoothLength
 Smoothing filter lenght. More...
 

Additional Inherited Members

- Static Public Attributes inherited from CVTracked
static AvgFloat trackingTimesMS
 Averaged time for video tracking in ms. More...
 
static AvgFloat detectTimesMS
 Averaged time for video feature detection & description in ms. More...
 
static AvgFloat detect1TimesMS
 Averaged time for video feature detection subpart 1 in ms. More...
 
static AvgFloat detect2TimesMS
 Averaged time for video feature detection subpart 2 in ms. More...
 
static AvgFloat matchTimesMS
 Averaged time for video feature matching in ms. More...
 
static AvgFloat optFlowTimesMS
 Averaged time for video feature optical flow tracking in ms. More...
 
static AvgFloat poseTimesMS
 Averaged time for video feature pose estimation in ms. More...
 
- Protected Attributes inherited from CVTracked
bool _isVisible
 Flag if marker is visible. More...
 
bool _drawDetection
 Flag if detection should be drawn into image. More...
 
CVMatx44f _objectViewMat
 view transformation matrix More...
 
HighResTimer _timer
 High resolution timer. More...
 

Detailed Description

OpenCV face & facial landmark tracker class derived from CVTracked.

Tracking class for face and facial landmark tracking. The class uses the OpenCV face detection algorithm from Viola-Jones to find all faces in the image and the facial landmark detector provided in cv::facemark. For more details see the comments in CVTrackedFaces::track method.

Definition at line 34 of file CVTrackedFaces.h.

Constructor & Destructor Documentation

◆ CVTrackedFaces()

CVTrackedFaces::CVTrackedFaces ( string  faceClassifierFilename,
string  faceMarkModelFilename,
int  smoothLength = 5 
)
explicit

Constructor for the facial landmark tracker.

The Constructor loads the training files for the face and the facial landmarks. It also defines an averaged set of facial landmark points in 3D used for pose estimation.

Parameters
faceClassifierFilenameName of the cascaded face training file
faceMarkModelFilenameName of the facial landmark training file
smoothLengthLength of averaging filter

Definition at line 31 of file CVTrackedFaces.cpp.

34 {
35  // Load Haar cascade training file for the face detection
36  if (!Utils::fileExists(faceClassifierFilename))
37  {
38  string msg = "CVTrackedFaces: File not found: " + faceClassifierFilename;
39  Utils::exitMsg("SLProject", msg.c_str(), __LINE__, __FILE__);
40  }
41  _faceDetector = new CVCascadeClassifier(faceClassifierFilename);
42 
43  // Load facemark model file for the facial landmark detection
44  if (!Utils::fileExists(faceMarkModelFilename))
45  {
46  string msg = "CVTrackedFaces: File not found: " + faceMarkModelFilename;
47  Utils::exitMsg("SLProject", msg.c_str(), __LINE__, __FILE__);
48  }
49 
50  cv::face::FacemarkLBF::Params facemarkParams;
51  facemarkParams.verbose = false; // no logging
52  _facemark = cv::face::FacemarkLBF::create(facemarkParams);
53  _facemark->loadModel(faceMarkModelFilename);
54 
55  // Init averaged 2D facial landmark points
56  _smoothLength = smoothLength;
57  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Nose tip
58  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Nose hole left
59  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Nose hole right
60  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Left eye left corner
61  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Left eye right corner
62  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Right eye left corner
63  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Right eye right corner
64  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Left mouth corner
65  _avgPosePoints2D.emplace_back(AvgCVVec2f(smoothLength, CVVec2f(0, 0))); // Right mouth corner
66 
67  _cvPosePoints2D.resize(_avgPosePoints2D.size(), CVPoint2f(0, 0));
68 
69  // Set 3D facial points in mm
70  _cvPosePoints3D.push_back(CVPoint3f(.000f, .000f, .000f)); // Nose tip
71  _cvPosePoints3D.push_back(CVPoint3f(-.015f, -.005f, -.018f)); // Nose hole left
72  _cvPosePoints3D.push_back(CVPoint3f(.015f, -.005f, -.018f)); // Nose hole right
73  _cvPosePoints3D.push_back(CVPoint3f(-.047f, .041f, -.036f)); // Left eye left corner
74  _cvPosePoints3D.push_back(CVPoint3f(-.019f, .041f, -.033f)); // Left eye right corner
75  _cvPosePoints3D.push_back(CVPoint3f(.019f, .041f, -.033f)); // Right eye left corner
76  _cvPosePoints3D.push_back(CVPoint3f(.047f, .041f, -.036f)); // Right eye right corner
77  _cvPosePoints3D.push_back(CVPoint3f(-.025f, -.035f, -.036f)); // Left Mouth corner
78  _cvPosePoints3D.push_back(CVPoint3f(.025f, -.035f, -.036f)); // Right mouth corner
79 }
Utils::Averaged< CVVec2f > AvgCVVec2f
cv::Point3f CVPoint3f
Definition: CVTypedefs.h:45
cv::Point2f CVPoint2f
Definition: CVTypedefs.h:43
cv::CascadeClassifier CVCascadeClassifier
Definition: CVTypedefs.h:66
cv::Vec2f CVVec2f
Definition: CVTypedefs.h:50
int _smoothLength
Smoothing filter lenght.
cv::Ptr< CVFacemark > _facemark
Facial landmarks detector smart pointer.
CVCascadeClassifier * _faceDetector
Viola-Jones face detector.
CVVPoint2f _cvPosePoints2D
vector of OpenCV point2D
vector< AvgCVVec2f > _avgPosePoints2D
vector of averaged facial landmark 2D points
CVVPoint3f _cvPosePoints3D
vector of OpenCV point2D
bool fileExists(const string &pathfilename)
Returns true if a file exists.
Definition: Utils.cpp:897
void exitMsg(const char *tag, const char *msg, const int line, const char *file)
Terminates the application with a message. No leak checking.
Definition: Utils.cpp:1135

◆ ~CVTrackedFaces()

CVTrackedFaces::~CVTrackedFaces ( )

Definition at line 81 of file CVTrackedFaces.cpp.

82 {
83  delete _faceDetector;
84 }

Member Function Documentation

◆ delaunayTriangulate()

void CVTrackedFaces::delaunayTriangulate ( CVMat  imageBgr,
const CVVPoint2f points,
bool  drawDetection 
)
static

Returns the Delaunay triangulation on the points within the image

Parameters
imageBgrOpenCV BGR image
points2D points as OpenCV vector of points 2D
drawDetectionFlag if detection should be drawn

Definition at line 241 of file CVTrackedFaces.cpp.

244 {
245  // Get rect of image
246  CVSize size = imageBgr.size();
247  CVRect rect(0, 0, size.width, size.height);
248 
249  // Create an instance of Subdiv2D
250  cv::Subdiv2D subdiv(rect);
251 
252  // Do Delaunay triangulation for the landmarks
253  for (const CVPoint2f& point : points)
254  if (rect.contains(point))
255  subdiv.insert(point);
256 
257  // Add additional points in the corners and middle of the sides
258  float w = (float)size.width;
259  float h = (float)size.height;
260  subdiv.insert(CVPoint2f(0.0f, 0.0f));
261  subdiv.insert(CVPoint2f(w * 0.5f, 0.0f));
262  subdiv.insert(CVPoint2f(w - 1.0f, 0.0f));
263  subdiv.insert(CVPoint2f(w - 1.0f, h * 0.5f));
264  subdiv.insert(CVPoint2f(w - 1.0f, h - 1.0f));
265  subdiv.insert(CVPoint2f(w * 0.5f, h - 1.0f));
266  subdiv.insert(CVPoint2f(0.0f, h - 1.0f));
267  subdiv.insert(CVPoint2f(0.0f, h * 0.5f));
268 
269  // Draw Delaunay triangles
270  if (drawDetection)
271  {
272  vector<cv::Vec6f> triangleList;
273  subdiv.getTriangleList(triangleList);
274  CVVPoint pt(3);
275 
276  for (auto t : triangleList)
277  {
278  pt[0] = CVPoint(cvRound(t[0]), cvRound(t[1]));
279  pt[1] = CVPoint(cvRound(t[2]), cvRound(t[3]));
280  pt[2] = CVPoint(cvRound(t[4]), cvRound(t[5]));
281 
282  // Draw rectangles completely inside the image.
283  if (rect.contains(pt[0]) && rect.contains(pt[1]) && rect.contains(pt[2]))
284  {
285  line(imageBgr, pt[0], pt[1], cv::Scalar(255, 255, 255), 1, cv::LINE_AA, 0);
286  line(imageBgr, pt[1], pt[2], cv::Scalar(255, 255, 255), 1, cv::LINE_AA, 0);
287  line(imageBgr, pt[2], pt[0], cv::Scalar(255, 255, 255), 1, cv::LINE_AA, 0);
288  }
289  }
290  }
291 }
cv::Rect CVRect
Definition: CVTypedefs.h:39
cv::Size CVSize
Definition: CVTypedefs.h:55
cv::Point CVPoint
Definition: CVTypedefs.h:41
vector< cv::Point > CVVPoint
Definition: CVTypedefs.h:75
bool drawDetection()
Definition: CVTracked.h:64

◆ track()

bool CVTrackedFaces::track ( CVMat  imageGray,
CVMat  imageBgr,
CVCalibration calib 
)
finalvirtual

Tracks the a face and its landmarks.

Implements CVTracked.

Definition at line 100 of file CVTrackedFaces.cpp.

103 {
105 
106  assert(!imageGray.empty() && "ImageGray is empty");
107  assert(!imageBgr.empty() && "ImageBGR is empty");
108  assert(!calib->cameraMat().empty() && "Calibration is empty");
109 
110  //////////////////
111  // Detect Faces //
112  //////////////////
113 
114  float startMS = _timer.elapsedTimeInMilliSec();
115 
116  // Detect faces
117  CVVRect faces;
118  int min = (int)((float)imageGray.rows * 0.4f); // the bigger min the faster
119  int max = (int)((float)imageGray.rows * 0.8f); // the smaller max the faster
120  CVSize minSize(min, min);
121  CVSize maxSize(max, max);
122  _faceDetector->detectMultiScale(imageGray,
123  faces,
124  1.05,
125  3,
126  0,
127  minSize,
128  maxSize);
129 
130  // Enlarge the face rect at the bottom to cover also the chin
131  for (auto& face : faces)
132  face.height = (int)(face.height * 1.2f);
133 
134  float time2MS = _timer.elapsedTimeInMilliSec();
135  CVTracked::detect1TimesMS.set(time2MS - startMS);
136 
137  //////////////////////
138  // Detect Landmarks //
139  //////////////////////
140 
141  CVVVPoint2f lm;
142  bool foundLandmarks = _facemark->fit(imageBgr, faces, lm);
143 
144  float time3MS = _timer.elapsedTimeInMilliSec();
145  CVTracked::detect2TimesMS.set(time3MS - time2MS);
146  CVTracked::detectTimesMS.set(time3MS - startMS);
147 
148  if (foundLandmarks)
149  {
150  for (unsigned long i = 0; i < lm.size(); i++)
151  {
152  // Landmark indexes from
153  // https://cdn-images-1.medium.com/max/1600/1*AbEg31EgkbXSQehuNJBlWg.png
154  _avgPosePoints2D[0].set(CVVec2f(lm[i][30].x, lm[i][30].y)); // Nose tip
155  _avgPosePoints2D[1].set(CVVec2f(lm[i][31].x, lm[i][31].y)); // Nose hole left
156  _avgPosePoints2D[2].set(CVVec2f(lm[i][35].x, lm[i][35].y)); // Nose hole right
157  _avgPosePoints2D[3].set(CVVec2f(lm[i][36].x, lm[i][36].y)); // Left eye left corner
158  _avgPosePoints2D[4].set(CVVec2f(lm[i][39].x, lm[i][39].y)); // Left eye right corner
159  _avgPosePoints2D[5].set(CVVec2f(lm[i][42].x, lm[i][42].y)); // Right eye left corner
160  _avgPosePoints2D[6].set(CVVec2f(lm[i][45].x, lm[i][45].y)); // Right eye right corner
161  _avgPosePoints2D[7].set(CVVec2f(lm[i][48].x, lm[i][48].y)); // Left mouth corner
162  _avgPosePoints2D[8].set(CVVec2f(lm[i][54].x, lm[i][54].y)); // Right mouth corner
163 
164  // Convert averaged 2D points to OpenCV points2d
165  for (unsigned long p = 0; p < _avgPosePoints2D.size(); p++)
166  _cvPosePoints2D[p] = CVPoint2f(_avgPosePoints2D[p].average()[0],
167  _avgPosePoints2D[p].average()[1]);
168 
169  // delaunayTriangulate(imageBgr, landmarks[i], drawDetection);
170 
171  ///////////////////
172  // Visualization //
173  ///////////////////
174 
175  if (_drawDetection)
176  {
177  // Draw rectangle of detected face
178  cv::rectangle(imageBgr,
179  faces[i],
180  cv::Scalar(255, 0, 0),
181  2);
182 
183  // Draw detected landmarks
184  for (auto& j : lm[i])
185  cv::circle(imageBgr,
186  j,
187  2,
188  cv::Scalar(0, 0, 255),
189  -1);
190 
191  // Draw averaged face points used for pose estimation
192  for (unsigned long p = 0; p < _avgPosePoints2D.size(); p++)
193  cv::circle(imageBgr,
194  _cvPosePoints2D[p],
195  5,
196  cv::Scalar(0, 255, 0),
197  1);
198  }
199 
200  // Do pose estimation for the first face found
201  if (i == 0)
202  {
203  /////////////////////
204  // Pose Estimation //
205  /////////////////////
206 
207  startMS = _timer.elapsedTimeInMilliSec();
208 
209  // find the camera extrinsic parameters (rVec & tVec)
210  CVMat rVec; // rotation angle vector as axis (length as angle)
211  CVMat tVec; // translation vector
212  bool solved = solvePnP(CVMat(_cvPosePoints3D),
214  calib->cameraMat(),
215  calib->distortion(),
216  rVec,
217  tVec,
218  false,
219  cv::SOLVEPNP_EPNP);
220 
222 
223  if (solved)
224  {
225  _objectViewMat = createGLMatrix(tVec, rVec);
226  return true;
227  }
228  }
229  }
230  }
231 
232  return false;
233 }
cv::Mat CVMat
Definition: CVTypedefs.h:38
vector< cv::Rect > CVVRect
Definition: CVTypedefs.h:74
vector< vector< cv::Point2f > > CVVVPoint2f
Definition: CVTypedefs.h:96
#define PROFILE_FUNCTION()
Definition: Instrumentor.h:41
const CVMat & cameraMat() const
const CVMat & distortion() const
CVMatx44f _objectViewMat
view transformation matrix
Definition: CVTracked.h:93
static AvgFloat detectTimesMS
Averaged time for video feature detection & description in ms.
Definition: CVTracked.h:83
bool _drawDetection
Flag if detection should be drawn into image.
Definition: CVTracked.h:92
static cv::Matx44f createGLMatrix(const CVMat &tVec, const CVMat &rVec)
Create an OpenGL 4x4 matrix from an OpenCV translation & rotation vector.
Definition: CVTracked.cpp:46
static AvgFloat detect1TimesMS
Averaged time for video feature detection subpart 1 in ms.
Definition: CVTracked.h:84
HighResTimer _timer
High resolution timer.
Definition: CVTracked.h:94
static AvgFloat detect2TimesMS
Averaged time for video feature detection subpart 2 in ms.
Definition: CVTracked.h:85
static AvgFloat poseTimesMS
Averaged time for video feature pose estimation in ms.
Definition: CVTracked.h:88
float elapsedTimeInMilliSec()
Definition: HighResTimer.h:38
void set(T value)
Sets the current value in the value array and builds the average.
Definition: Averaged.h:53

Member Data Documentation

◆ _avgPosePoints2D

vector<AvgCVVec2f> CVTrackedFaces::_avgPosePoints2D
private

vector of averaged facial landmark 2D points

Definition at line 53 of file CVTrackedFaces.h.

◆ _boundingRect

CVRect CVTrackedFaces::_boundingRect
private

Bounding rectangle around landmarks.

Definition at line 54 of file CVTrackedFaces.h.

◆ _cvPosePoints2D

CVVPoint2f CVTrackedFaces::_cvPosePoints2D
private

vector of OpenCV point2D

Definition at line 55 of file CVTrackedFaces.h.

◆ _cvPosePoints3D

CVVPoint3f CVTrackedFaces::_cvPosePoints3D
private

vector of OpenCV point2D

Definition at line 56 of file CVTrackedFaces.h.

◆ _faceDetector

CVCascadeClassifier* CVTrackedFaces::_faceDetector
private

Viola-Jones face detector.

Definition at line 51 of file CVTrackedFaces.h.

◆ _facemark

cv::Ptr<CVFacemark> CVTrackedFaces::_facemark
private

Facial landmarks detector smart pointer.

Definition at line 52 of file CVTrackedFaces.h.

◆ _smoothLength

int CVTrackedFaces::_smoothLength
private

Smoothing filter lenght.

Definition at line 57 of file CVTrackedFaces.h.


The documentation for this class was generated from the following files: