Using Intel's MKL for Deep Neural Nets, Curie Modules, and Real Sense 3D Cameras To Capture The ArtistRy behind Brushstrokes


 


Using Beats and elasticsearch to model Artistic Style

Have been making painting robots for about ten years now. These robots use a variety of AI and Machine Learning approaches to paint original compositions with an artist’s brush on stretched canvas. The paintings we create together have gotten a lot of attention recently including awards and recognition from Google, TEDx, Microsoft, The Barbican, SXSL at The White House, robotart.org, and NPR. You can see more exhibitions and press write ups at www.cloudpainter.com/press though the quickest way to get up to speed is to check out this video.

In spite of all our recent success however, there is a major systematic problem with the artwork we are creating. While the images are aesthetically pleasing, the individual brushstrokes themselves lack artistic style. They are noticeably mechanical and repetitive.

For the next iteration of our painting robot, We have found a way to teach it style. It will records detailed metrics of a paintbrush being used by an artist to create a painting then use deep learning algorithms to model the style of those brushstrokes. A robotic arm will then use the model to create pastiches, new original compositions in the style of the artist.

To get into technical details, this will be done by embedding Intel’s Curie Module into a set of paintbrushes. These will be capable of recording detailed information of the brushstrokes movement in time and space. Multiple Intel Real Sense cameras will also be set up facing a blank canvas. An artist will then be invited to paint an image on the canvas. Accelerometers on the Curie module will detect and record all of the brushes movements and the Real Sense cameras will detect and record all of the marks made on the canvas.  All this data will be forwarded by Beats into an Elasticsearch database. When the artist completes the painting, Deep Learning will be leveraged to process the detailed brushstroke data into a model. This brushstroke model will then be used to direct a robot arm as it attempts to create its own original painting in the style of how the brush was used. The subject of the robotic arm's painting can be anything, though ideally it would from an original image taken by one of the Real Sense cameras. Perhaps a portrait of a face that it detects?

Hopefully the robot is able to learn to paint in the artistic style of the artist. 
 

Deep Learning Approach

This project will rely heavily on recent state-of-the-art research by Google Brain and advances made in Microsoft's COCO Project (Common Objects in Context.)  Over the past year, Google’ has been refining a technique where deep learning algorithms are “painting” photographs in the style of a famous artists. If you have not seen one of Google's pastiches, here is an example taken from their blog of several photos painted in the style of various famous artists.

Taken from https://research.googleblog.com/2016/10/supercharging-style-transfer.html

Taken from https://research.googleblog.com/2016/10/supercharging-style-transfer.html

The results are amazing, but when you look more closely you see a systematic problem that is similar to the problem our robots have. 

static1.squarespace-3.jpg

While Vincent Dumoulin, Jonathon Shlens, and Majunath Kudlar of the Google Brain Team have done an amazing job of transferring the color and texture from the original painting, they did not really capture the context or the style of the brushstrokes. I offer this critique despite being deeply indebted to their work and a big fan of what they are accomplishing.

In regards to confusing the context, Brad Pitt's forehead and woman's face is painted in the style of the overcast sky from the original painting.  The pastiche really has no idea that these areas were faces and not a sky.  We think we can improve upon this misconception by leveraging recent advances in Microsoft's COCO Project where contestants have been refining the art of identifying common objects in context.  Furthermore we believe that using the Real Sense 3D Camera will aid greatly in separating foregrounds from backgrounds and likewise prevent this stylistic problem. Leveraging the 3D and simple face detection algorithms, it is quite easy to accomplish this.

The second problem with these pastiches, is they don't have any brush stroke data. For example, neither face has the swooping strokes of the face in Munch's The Scream. Furthermore, The Golden Gate bridge is overly detailed when in the painting it is composed of long stark dry strokes. Munch's brushstroke is not truly understood or transferred. This is a problem because ultimately The Scream is a painting where the style of the brushstroke is a major contributor to the aesthetic effect.

 

Capturing the Human Element

As mentioned previously Deep Learning is at the center of how the context and brushstroke problems described above will be solved.  While the technique of creating pastiches being promoted by Google uses only a single source image and attempts to paint new composition in that style, we will be using three sources of data in our deep neural network.

1: The original source image, exactly how Google already is.
2: A Brushstroke Map made with help from Curie Module and Real Sense 3D Camera
3: An Context Map of the source painting and new photo pastiche is being created for

 

By combining these three data sources it is believed that true pastiches can be made from famous paintings that are true to both the brushstroke and context of the original artist.  For example in the portraits above, the red and orange sky would only behind the heads and above the bridge.

Even more interesting than transferring the style of famous painters is that the brushstrokes can come from anyone and the source image can be anything, like a simple picture of a room for example.  Imagine if the source image is a room, and the brushstroke model was made by recording a child playing with paint on a piece of construction paper.  The robot could then take a photo of the child's parent, and paint a realistic portrait with random looking strokes that matches the style and decor of the room.  Visually it will be a realistic portrait, with emotional significance because child's brushstrokes were involved. And since the room was the source image, it would match the color and decor of the room. Its ready to hang. Just endless possibilities.

The beauty of all this for Intel, is that this can be done using MKL Deep Neaural Networks, the Curie Module, and the Real Sense 3D Camera, and our latest robotic painting machine, seen below.

static1.squarespace-4.jpg

I have no idea where this project will lead or end. I gave a TEDx Talk on Artificial Creativity 6 months ago and already feel the talk is obsolete.  By the time we get this project up and running in next couple of months, there are sure to be a couple new AI developments that I would want to explore.

Will leave you with a video of our stroke mapping for Vincent Van Gogh's Starry Night, for no other reason than the fact that it looks cool.

 

Pindar Van Arman