[C#] VGG Face - Multiple face detection / recognition

Last time we tried VGG Face at previous post to display face recognition in one picture, this time we improve it, detect multiple face in one image and recognize them.

Here I will use Haar Cascades to do face detection, you can also download other feature-based cascade classifiers at here.



Let's see the code, most different part was here.

var faceCascade = new CascadeClassifier();
faceCascade.Load(cascade);
var faces = faceCascade.DetectMultiScale(org, 1.1, 6, HaarDetectionType.DoRoughSearch, new Size(60, 60));
var faceList = new List();
foreach (var rect in faces)
{
    Cv2.Rectangle(org,rect,Scalar.Red);
    faceList.Add(org[rect]);
}

We using Haar Cascades - Face to get face location, and crop it as a list for later use.


Result display part has a little difference too.

for (int n = 0; n < prob.Height; n++)
{
    //convert result to list
    var probList = new Dictionary();
    for (int i = 0; i < prob.Width; i++)
    {
        probList.Add(i, prob.At(n, i));
    }

    //get top 1
    var top1 = probList.OrderByDescending(x => x.Value).First();
    var label = $"{labels[top1.Key]}:{top1.Value * 100:0.00}%";
    Console.WriteLine(label);

   //show if confidence > 50%
   if (top1.Value > 0.5)
   {
       var textsize = Cv2.GetTextSize(label, HersheyFonts.HersheyTriplex, 0.5, 1, out var baseline);
       var y = faces[n].TopLeft.Y - textsize.Height - baseline <= 0
             ? faces[n].BottomRight.Y + textsize.Height + baseline : faces[n].TopLeft.Y - baseline;
       //draw result
       org.PutText(label, new Point(faces[n].TopLeft.X,y), HersheyFonts.HersheyTriplex, 0.5, Scalar.OrangeRed);
    }
}

First for-loop n<prob.Height was for how many faces.
2nd for-loop i<prob.Width was for 2622 identities.
This time we draw name on image only if confidence over 50%, because if we input face not in VGG Face Dataset, the confidence should be low, so we skip it.

If we don't skip them, you will see wrong name with low confidence like that.


And here is the test result.


In my opinion, those two test shows how to detection face in image and recognize them, if you want to implement it into surveillance system should be not so hard, I know it need more work to make it better, but at least it is a good start.

BTW, If you think 2622 identities not enough, you can try VGGFace2 Dataset, it got 9131 identities.

---

Here is the full code.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using OpenCvSharp;
using OpenCvSharp.Dnn;

namespace VggFaceCrop
{
    class Program
    {
        static void Main()
        {
            var file = "best-supporting-actress.jpg";
            //var file = "oscars-2017.jpg";
            var prototxt = "VGG_FACE_deploy.prototxt";
            var model = "VGG_FACE.caffemodel";
            var labeltxt = "names.txt";
            var cascade = "haarcascade_frontalface_default.xml";
            var org = Cv2.ImRead(file);

            //get face using haarcascades , https://github.com/opencv/opencv/tree/master/data/haarcascades
            var faceCascade = new CascadeClassifier();
            faceCascade.Load(cascade);
            var faces = faceCascade.DetectMultiScale(org, 1.1, 6, HaarDetectionType.DoRoughSearch, new Size(60, 60));
            var faceList = new List();
            foreach (var rect in faces)
            {
                Cv2.Rectangle(org,rect,Scalar.Red);
                faceList.Add(org[rect]);
            }

            //read all names
            var labels = ReadLabels(labeltxt);
            var blob = CvDnn.BlobFromImages(faceList, 1, new Size(224, 224));
            var net = CvDnn.ReadNetFromCaffe(prototxt, model);
            net.SetInput(blob, "data");

            Stopwatch sw = new Stopwatch();
            sw.Start();
            //forward model
            var prob = net.Forward("prob");
            sw.Stop();
            Console.WriteLine($"Runtime:{sw.ElapsedMilliseconds} ms");

            for (int n = 0; n < prob.Height; n++)
            {
                //convert result to list
                var probList = new Dictionary();
                for (int i = 0; i < prob.Width; i++)
                {
                    probList.Add(i, prob.At(n, i));
                }

                //get top 1
                var top1 = probList.OrderByDescending(x => x.Value).First();
                var label = $"{labels[top1.Key]}:{top1.Value * 100:0.00}%";
                Console.WriteLine(label);

                //show if confidence > 50%
                if (top1.Value > 0.5)
                {
                    var textsize = Cv2.GetTextSize(label, HersheyFonts.HersheyTriplex, 0.5, 1, out var baseline);
                    var y = faces[n].TopLeft.Y - textsize.Height - baseline <= 0
                        ? faces[n].BottomRight.Y + textsize.Height + baseline : faces[n].TopLeft.Y - baseline;
                    //draw result
                    org.PutText(label, new Point(faces[n].TopLeft.X,y), HersheyFonts.HersheyTriplex, 0.5, Scalar.OrangeRed);
                }
            }
            using (new Window("image", org))
            {
                Cv2.WaitKey();
            }
        }

        // Load label name list
        public static string[] ReadLabels(string file)
        {
            const string patten = "(?.*)\n";
            var result = new List();
            using (FileStream stream = File.OpenRead(file))
            using (StreamReader reader = new StreamReader(stream))
            {
                string text = reader.ReadToEnd();
                if (string.IsNullOrWhiteSpace(text))
                {
                    return result.ToArray();
                }
                Regex regex = new Regex(patten);
                var matches = regex.Matches(text);
                result.AddRange(from Match match in matches select match.Groups[1].Value);
                return result.ToArray();
            }
        }
    }
}

Or you can get it from my github.

Hope you enjoy it.