chadiik - Sketches of The Obsessive Drafter

chady karlitch

Software & multimedia

call me:

email me:

Contact

TOP ↑

Sketches of The Obsessive Drafter

Facial capture and generative sketches

The Obsessive Drafter is a large autonomous robot arm that interpretively draws portraits of the people that approach him. Initiated and led by Guillaume Credoz who created the mechanical structure, and completed by collaborations with Nareg Karaoghlanian for mechatronics including mobility program, and myself for the sketch generation process.

Saint-Étienne Design Biennial 2017

March 9 until April 9 2017, Building H - Cité du design: TOD was exhibited as part of "Si automatique?" . During the month-long performance, it was to fill a 6 by 3 meters wall with sketched portraits of visitors…

Photo: Panorama of 'Cité du design'

Sketch generation process

Using C# (visual studio 2013) and EMGU, an OpenCV wrapper.

The software will automatically capture the faces of visitors, through a webcam attached at TOD's wrist, then convert the image into a short sketch for feeding to the machine.

1) Computer Vision

I didn’t train my own sets, but used the default OpenCV eyes and faces haar cascade

OpenCV Face Detection Visualized - Adam Harvey

definitions that comes with the installation: A basic usage of those definitions alone returned a lot of false-positives .

Some filters were created:

Min & max dimensions
Also used for ensuring a min image quality and that hair, ears and neck would not be out of capture feed bounds.
Mobility range check
This would get rid of stationary false-positives that usually get extracted from the environment and of fast moving people whose image would turn out blurry.
Growth and shrinkage limits
Video noise had a tendency to agglomerate into larger and larger reported blobs.
Capture time
Filters out captures that are less than 1 second. It also ensures a better capture quality.
Face tracking
Ensures that a head is captured once, unless it goes out of frame then back in. (Not by facial recognition, but by tracking the capture's rectangle)

Here’s a capture testing session, against a 46 min documentary :

A face being tracked has a score

float score = area * properties.dimensionsScoreMul + time * properties.timeScoreMul + eyes * properties.eyesScoreMul;
Variables are limited (ie, eyes:[0, 2]) and the highest possible overall score is 1.
For a 0.25 eyes score multiplier, a one-eyed jack would usually lose against a two-eyed queen, even if there capture dimensions where a bit smaller.

that is updated each frame.

This score is then used while pooling for a portrait to sketch, including a final additional time filter: As the performance last a month, we initially intended for some of the captured visitors to still be around when their portraits are being sketched on the wall.

2) Sketch generation

The sketch routine was suggested to be a continuous circling line with variable diameter depending on the brightness of the underlying area.

The steering behavior of the pen tip was to be defined.

A quick prototype was created in AS3:
A wandering particle that is attracted by the color proximity of neighboring areas, in turn this particle influence the underlying area by progressively fading it out of existence.

Some parameters returned a cool (inefficient) contour tracer.

Gallery of results from initial prototype, using automated parameters space exploration:

New creatures
Parameters exploration
Parameters exploration

Another approach

A couple of different trials later, I decided to stop playing and define specific rules: Normalize and filter the capture image, create a dithered version, order the dither points to generate a path, detailing the picture along it.

Image filtering:

Noise removal
Non-local means denoising: FastNlMeansDenoisingColored
Masking against a pre-generated ‘face focus’ map.
Image histogram equalization, using CLAHE .
Minimizing large black patches
By masking with a dilated edges extraction (Edges obtained by a gradient convolution kernel, using MorphologyEx

Edges are also extracted, using Hough transform, and were intended to add additional detailing around the center of the face: eyes, nose, mouth. But were not ultimately used.

Dither map:

Used Floyd–Steinberg dithering.
Probability of dither points based on distance from a central circle (fading out).

Ordering:

The line would have as many breaks as needed, otherwise it’s a traveling salesman problem!

Trial 1, minimum spanning tree:

Delaunay triangulation used to compute the euclidean minimum spanning tree of the dither points:

Trial 2, (simpler is better) weighted sort:

Weighted by distance (between vertices), hue, brightness and proximity to center. This will tend to build up the face in quasi-sequence by facial feature.

Each path node is encircled once, and the path can break when distances are above a threshold.

Below is one the final results, an unavoidable sketching the lovely Lena Soderberg:

Wall filling & scenario:

A completed sketch is a normalized turtle graphic. It’s transformed to fit the next available cell on wall according to a running scenario, which data are written to disk.

The entire process runs on a windows laptop, the drawing commands get sent to the Arduino via serial communication.

Machine execution

The wall at the end of the month:

Photo by Guillaume Credoz.

Another project featuring TOD's avatar: https://chadiik-com-caa17.web.app/?article=PublicCanvas

Link https://chadiik-com-caa17.web.app/articles/sketchesoftod/ Go to the comments pool.