A hands-on look at the Amazon Rekognition API

Amazon Rekognition is a Deep Learning based image analysis service. Don’t worry though, you won’t have to wade through Machine Learning / Deep Learning mumbo jumbo to work with Recognition. Quite the contrary, as Rekognition provides a very easy-to-use API. 
 
 It allows developers to:

• detect thousands of objects and scenes;
• analyze faces;
• compare two faces to measure similarity;
• build face collections and match faces against these collections.

As usual, this service can be used with the AWS CLI (as in ‘aws rekognition’ ), or with one of our language SDKs. I’ll show you some CLI examples first and then we’ll use the popular Python SDK, aka boto3.

First things first: how do we send images for processing? Two options: send the image as a byte blob or put it in S3. I suspect the most of use will use the second option, so that’s what I’ll use. Time to play!

$ aws rekognition detect-faces --image "S3Object={Bucket="jsimon-public", Name="julien1.jpg"}"
 
{ "FaceDetails": [ { "BoundingBox": { "Width": 0.3883333206176758, "Top": 0.12222222238779068, "Left": 0.33666667342185974, "Height": 0.2588889002799988 }, "Landmarks": [ { "Y": 0.23426248133182526, "X": 0.46131378412246704, "Type": "eyeLeft" }, { "Y": 0.22791674733161926, "X": 0.5936729311943054, "Type": "eyeRight" }, { "Y": 0.27828338742256165, "X": 0.5404868721961975, "Type": "nose" }, { "Y": 0.3229646682739258, "X": 0.48395034670829773, "Type": "mouthLeft" }, { "Y": 0.31654009222984314, "X": 0.5957114696502686, "Type": "mouthRight" } ], "Pose": { "Yaw": 4.216298580169678, "Roll": -4.777482509613037, "Pitch": -2.406636953353882 }, "Quality": { "Sharpness": 70.0, "Brightness": 65.17163848876953 }, "Confidence": 99.99468231201172 } ], "OrientationCorrection": "ROTATE_0" }

JSON, the cornerstone of any nutritious service. So, what do we have here? A face has been found with 99.99+% confidence. It’s delimited by the BoundingBox coordinates (top left corner, face width, face height): these are fractional values with respect to the total height and width of the image. Eyes, nose and mouth have been located too (that’s reassuring).

Now, let’s see what Rekognition can tell us about this second picture.

$ aws rekognition detect-labels --image '{"S3Object":{"Bucket":"jsimon-public","Name":"julien2.jpg"}}'

{ "Labels": [ { "Confidence": 99.29261779785156, "Name": "Human" }, { "Confidence": 99.2958984375, "Name": "People" }, { "Confidence": 99.2958984375, "Name": "Person" }, { "Confidence": 99.2667007446289, "Name": "Book" }, { "Confidence": 99.2667007446289, "Name": "Text" }, { "Confidence": 71.22590637207031, "Name": "Bookcase" }, { "Confidence": 71.22590637207031, "Name": "Furniture" }, { "Confidence": 71.22590637207031, "Name": "Shelf" }, { "Confidence": 52.00172805786133, "Name": "Portrait" }, { "Confidence": 52.00172805786133, "Name": "Selfie" } ] }

With a very good level of confidence, this is the picture of a human with books on a bookshelf, possibly a portrait. A pretty good summary. Let's compare the two previous pictures. Is this truly the same person? Spoiler: yes, although I look 15 years older on the first one. Note to self: no more promo shots after 36 sleepless hours :D

$ aws rekognition compare-faces --source-image '{"S3Object":{"Bucket":"jsimon-public","Name":"julien1.jpg"}}' --target-image '{"S3Object":{"Bucket":"jsimon-public","Name":"julien2.jpg"}}

{ "FaceMatches": [ { "Face": { "BoundingBox": { "Width": 0.5596370100975037, "Top": 0.1318063884973526, "Left": 0.3889369070529938, "Height": 0.5596370100975037 }, "Confidence": 99.98912811279297 }, "Similarity": 98.0 } ], "SourceImageFace": { "BoundingBox": { "Width": 0.3883333206176758, "Top": 0.12222222238779068, "Left": 0.33666667342185974, "Height": 0.2588889002799988 }, "Confidence": 99.99468231201172 } }

Similarity is 98%. Jet lag or not, I'm always the same me. See how simple this service is? I don't see how they could have made it easier. How long would it take to design, build and *train* something like this on your own? I have really no idea and to I don't intend to find out!

Enough CLI, let’s switch to Python and run more visual examples. For this purpose, I’ve written a couple of scripts (available here), using boto3 and the Pillow image processing library.

In a nutshell:

• rekognitionDetect.py bucket_name image [copy | nocopy ] : try to detect faces inside an image. If faces are found, each of them will be highlighted by a box and an updated image will be saved. The script will also report image labels and face information (gender, beard, glasses, etc.). Maximum number of labels and default confidence are respectively set to 10 and 75% by default.
• rekognitionCompare.py bucket_name sourceImage targetImage [copy | nocopy ]: try to match a reference face to another image. If the face is found, it will be highlighted by a box and an updated image will be saved.

All images must be present with the same name both locally and in S3 . The last parameter for both scripts allows you to skip the copy to S3 if the file is already there. Hopefully, the code reads like well-written prose (hi Uncle Bob). If not, blame jet lag (yes, it’s the root of all evil). Anyway, there’s nothing complicated here, I’m sure you’ll figure it out in no time. Let’s play some more!

$ rekognitionDetect.py jsimon-public booth1.jpg nocopy

Label Human, confidence: 99.3180236816
 Label People, confidence: 99.3190917969
 Label Person, confidence: 99.3190917969
 Label Clothing, confidence: 92.1037216187
 Label Overcoat, confidence: 92.1037216187
 Label Suit, confidence: 92.1037216187
 Label Computer, confidence: 76.0058441162
 Label Electronics, confidence: 76.0058441162
 Label LCD Screen, confidence: 76.0058441162
 Label Laptop, confidence: 76.0058441162
 *** Face 0 detected, confidence: 99.999671936 Gender: Male HAPPY 96.4477920532 CALM 8.28260231018 CONFUSED 1.53788328171
 *** Face 1 detected, confidence: 99.9654922485 Gender: Male Beard Mustache HAPPY 98.5274353027 ANGRY 5.03668212891 CONFUSED 2.61067152023
 *** Face 2 detected, confidence: 99.9955444336 Gender: Male Eyeglasses HAPPY 97.6237945557 ANGRY 1.31589770317 CALM 0.939458608627
 *** Face 3 detected, confidence: 99.9996109009 Gender: Male Eyeglasses HAPPY 98.9962310791 SAD 11.4119710922 CONFUSED 1.69576406479

Say hi to Romain, Cédric and Damian, my friendly AWS colleagues. Rekognition sees 4 males, 1 with a beard, 2 with eyeglasses, all of them very happy... and I'm the calmest of the bunch, how about that. Amazingly, Rekognition manages to catch my hardly visible laptop (left edge of the picture, on the table).

Here’s a tougher one (Hallo to my German friends).

$ rekognitionDetect.py jsimon-public oktoberfest.jpg nocopy

output file
 
Label People, confidence: 99.0898742676
 Label Person, confidence: 99.0898971558
 Label Human, confidence: 99.0639343262
 Label Alcohol, confidence: 88.8537063599
 Label Beverage, confidence: 88.8537063599
 Label Drink, confidence: 88.8537063599
 Label Crowd, confidence: 84.0972671509
 Label Female, confidence: 84.0796279907
 Label Girl, confidence: 84.0796279907
 *** Face 0 detected, confidence: 99.9854202271 Gender: Male HAPPY 60.5386123657 ANGRY 12.2481765747 DISGUSTED 2.10083723068
 *** Face 1 detected, confidence: 99.9825744629 Gender: Female HAPPY 98.0062866211 SURPRISED 10.8561573029 SAD 0.810676813126
 *** Face 2 detected, confidence: 99.9904937744 Gender: Female HAPPY 84.5134887695 SURPRISED 8.68589305878 ANGRY 1.35719180107
 *** Face 3 detected, confidence: 99.9073257446 Gender: Male Beard Mustache HAPPY 80.5190963745 SURPRISED 23.9800624847 ANGRY 1.17569565773
 *** Face 4 detected, confidence: 99.9972229004 Gender: Male Mustache HAPPY 75.2949371338 CONFUSED 10.9511556625 DISGUSTED 1.91761255264
 *** Face 5 detected, confidence: 99.9999771118 Gender: Male HAPPY 35.9886474609 SURPRISED 3.75992059708 ANGRY 2.48707532883
 *** Face 6 detected, confidence: 99.9915084839 Gender: Female HAPPY 99.4766082764 CALM 0.791561603546 ANGRY 0.620931386948
 *** Face 7 detected, confidence: 99.9998931885 Gender: Female HAPPY 99.8826293945 SAD 7.21873044968 DISGUSTED 5.48685789108
 *** Face 8 detected, confidence: 83.6580963135 Gender: Male Eyeglasses SAD 94.9213943481 SURPRISED 76.9153442383 HAPPY 8.52976131439
 *** Face 9 detected, confidence: 99.9944610596 Gender: Male HAPPY 27.327457428 DISGUSTED 26.6790218353 ANGRY 12.1302127838
 *** Face 10 detected, confidence: 99.9998855591 Gender: Male SURPRISED 99.2624435425 HAPPY 22.0922241211 SAD 6.69546127319
 *** Face 11 detected, confidence: 99.9861831665 Gender: Male SURPRISED 60.7816810608 SAD 7.07310438156 HAPPY 3.66672611237
 *** Face 12 detected, confidence: 99.9990692139 Gender: Male HAPPY 48.0631027222 SURPRISED 2.61369943619 CONFUSED 2.40399837494
 *** Face 13 detected, confidence: 87.6368408203 Gender: Male HAPPY 16.2307357788 SAD 14.2565965652 ANGRY 12.3210906982
 *** Face 14 detected, confidence: 99.9553375244 Gender: Male HAPPY 54.3005943298 DISGUSTED 5.99133396149 SURPRISED 3.63597273827

Wow, 15 people, including partial faces. All genders are correct. Emotions are mostly ok, but we definitely need to add 'DRUNK' to the list ;) The labels are spot on: a crowd of men and women drinking alcohol. Let's try another one. Low res, low quality.

$ rekognitionDetect.py jsimon-public maradona.jpg nocopy
 
Label People, confidence: 99.2043991089
 Label Person, confidence: 99.2043991089
 Label Human, confidence: 99.1917037964
 Label Football, confidence: 97.2220993042
 Label Soccer, confidence: 97.2220993042
 Label Sport, confidence: 97.2220993042
 Label American Football, confidence: 83.3328475952
 Label Athlete, confidence: 78.3234786987
 *** Face 0 detected, confidence: 99.963470459 Gender: Male Mustache SURPRISED 21.8802871704 CALM 17.4065952301 SAD 11.6566238403
 *** Face 1 detected, confidence: 99.9813308716 Gender: Male Eyeglasses HAPPY 38.6969680786 ANGRY 6.79734945297 SURPRISED 2.61010527611
 *** Face 2 detected, confidence: 99.9385604858 Gender: Male SURPRISED 36.6970825195 SAD 7.66330337524 ANGRY 6.10639476776
 *** Face 3 detected, confidence: 99.9514923096 Gender: Male SAD 32.6836242676 DISGUSTED 4.55095767975 HAPPY 4.19711828232
 *** Face 4 detected, confidence: 99.8046951294 Gender: Male Beard Mustache SAD 46.0139579773 HAPPY 4.15547084808 DISGUSTED 0.981283187866
 *** Face 5 detected, confidence: 99.2888412476 Gender: Male SAD 90.2270889282 CALM 5.9303817749 HAPPY 3.26179981232

Labels are fine, except for 'American Football'. 83%??? Gimme a break, the training set needs more Soccer images! In addition, I don't think number 4 is wearing eyeglasses, but again this is a low res picture. Apart from this, Rekognition correctly picked up all faces and funny enough, the expressions make sense too: "sad" and "surprised" are definitely how these guys must have felt against the legendary Diego! A last one for the road: how about this complex abstract-ish nighttime picture of Shinjuku?

$ rekognitionDetect.py jsimon-public shinjuku.jpg nocopy

Label City, confidence: 88.4259796143 Label Downtown, confidence: 88.4259796143
 Label Metropolis, confidence: 84.8462677002
 Label Urban, confidence: 84.8462677002
 Label Night, confidence: 69.7816467285
 Label Outdoors, confidence: 69.7816467285
 Label Shop, confidence: 68.228477478
 Label Flyer, confidence: 60.3522796631
 Label Poster, confidence: 60.3522796631
 Label Neighborhood, confidence: 55.3994293213
 *** Face 0 detected, confidence: 97.9367828369
 Gender: Female SAD 46.1420478821 ANGRY 7.63346576691 HAPPY 6.28939962387

Note that I lowered the confidence threshold from 75% to 50% get more labels. Still, Rekognition does a good job. It also gets the girl's face and yes, she does look quite sad. The Anime face isn't detected but I guess this is the desired behavior. Alright, enough detection. Let's now try to match faces, using some of the previous pictures as well as some new ones.

$ rekognitionCompare.py jsimon-public julien1.jpg julien2.jpg nocopy
 Face match, confidence=99.9891281128, similarity=98.0
 
$ rekognitionCompare.py jsimon-public julien1.jpg booth1.jpg nocopy
 Face match, confidence=99.999671936, similarity=96.0
 
$ rekognitionCompare.py jsimon-public julien1.jpg booth2.jpg nocopy
 Face match, confidence=99.9991455078, similarity=84.0
 
$ rekognitionCompare.py jsimon-public julien1.jpg keynote.jpg no copy
 Face match, confidence=99.9932250977, similarity=82.0

Quite good! The last one is particularly nice, given the distance, the angle and the poor lighting (see actual picture above). These are just a few examples and I'm sure you can't wait to try your own. Hopefully this post has given you a visual, hands-on overview of the Recognition service and how user-friendly it is. I didn't cover face collections, but the API is pretty much what you'd expect (create, delete, etc.).

Feel free to explore and experiment. Until we meet again, keep rockin’.

Originally published at blog.julien.org on November 30, 2016.