GAN-Supported Concept Art Workflows

Preface

Introduction

Applications

Pitfalls

The Future
 

Preface

We all love the fact that computers can execute annoying work for us. Work we already know how to do, work that is repeatable and, often, also repetitive.

For the past few decades, new processes such as procedural generation have been helping us achieve diverse results with minimal input, leaving us to focus on being creative. Be it in the shape of procedural level generation of early rogue-like games, procedural nature such as Speedtree or, lately, the vast possibilities of procedural texturing with noise procedurality as seen in Substance Designer.

And now there’s a new kid in town: GAN’s - short for Generative Adversarial Networks. 

Neural networks that generate new data and in the case of so called StyleGAN’s it creates images or sequences.

These machine learning frameworks are making two AI’s play against each other to test and learn what would be considered to be a realistic result. This is based on the library you are feeding the network.

When we look back at the early versions of GAN’s, they were quite rudimentary and the results more than questionable. At least nobody would have said; “Let’s use this scary DeepDream image of a dog knight for our next big video game production.” 

(Credit Alexander Mordvintsev / Google DeepDream 2015 ©)

However, even during these clumsy beginnings, bigger companies were already seeing GAN’shuge potential. And now the time has come in which not only big companies are able to use neural networks to support their production, but it is also available, and affordable, for small studios, freelance artists and general consumers.

This change opens up huge opportunities. Endless ones. It also comes with some potential dangers and pitfalls that we as creatives should be aware of. In the following I will give an overview of how GAN’s can be successfully used, what the future might bring and what all this will mean for our job in the next 5 years.

Introduction

Let’s take a small trip back in time. This is an image I created one and a half years ago with a GAN website where you could input color-coded areas that defined the property of the environment. You could paint where you wanted clouds, a river, a wide mountain range or even a building with four rows of windows and a large door.

You could then click the generate button and “magic” would happen. I was stunned when I first tried it, as were most of  my colleagues and friends after I shared the results.

For the first time it felt like these networks had become so powerful that a production would actually benefit from them, instead of hours of work, this tool could easily spew out one realistic landscape concept after the other in mere minutes. At around the same time, a team at NVIDIA was working on their own tool which also allowed you to change the overall light mood and time of day alongside the segmentation map input I just mentioned.

https://www.youtube.com/watch?v=p5U4NgVGAwg

A couple of months later I saw a great talk by Scott Eaton who was building a deep-learning AI network and feeding it with sets of photos he shot of performers. He would use this self-built network to create abstract human shapes which used his line drawings as input. 

He eventually reached a point of experimentation where he trained the network on less figuratively modeled cubes and shapes. After the AI learned to interpret its new library, Scott took the results which, once again, were based on his linear drawing input and made them into physical sculptures .

https://www.youtube.com/watch?v=TN7Ydx9ygPo

Skip forward in time to about two months ago, the beginning of a new production cycle in our studio. Wonderful pre-production time. And as is often the case during early development, we had a little bit more freedom to experiment, to get back to the drawing board and find cool new ways to not only create crazy visuals but also re-think our art departments pipeline. Pre-production is always a good time to try and pinpoint time-consuming chores and how to overcome issues of terrible software bridging, or even just generally optimize everyone’s workflow by a bit.

With this in mind I thought of a website that I had used a while ago, called Ganbreeder. This page allowed you to input your own images and “cross-breed” them with existing images from other creators or its library. Since then the website was renamed to Artbreeder,  and now hosted a wide array of GAN’s trained for specific purposes such as environments, characters, faces or the more specialized categories like anime and furry heads.

https://artbreeder.com/create

So I took a real deep-dive into these generators.

It took me about two days of using the tool to stop being disturbed by my own creations. Don’t get me wrong, I loved the tool and quickly became addicted to it, but the results also gave mean eerie feeling of unease, of uncanny-ness sometimes. Once this period passed, I showed my results to the team and we we began discussing the possibilities this crossbreeder offered.

Applications

Let’s be real. If you were to properly set up and train your own neural network, the options you have are limitless. And you can create them much more specifically geared towards a certain purpose, not only by what library you train the system on, but also by how you want your input methods and variables to shape the output.

Realistically though, for most of us without a server farm in the basement or proper programming/scripting skills, we are limited to website options and fiddling around with the results we can get there.

For our production I found a couple of major aspects that already proved to be really helpful. Not only as time savers, but also to get new creative ideas in. Ideas that one might not imagine on their own.

Our project needed us to create concepts which felt really alien and unexpected, and so far this is one of the aspects in which GAN’s show their strength. They are able to deliver results which appear realistic at first glance, but can create weird,unusual shapes and designs if you allow them to.

Here is a break-down of several use cases for GAN’s and how I worked with them up until now.

Character Concepts

So far I have very mixed feelings with the current state of usability for character concepts. The reason being; Characters are the core of any film, game or any other storytelling medium. They are carefully crafted and rely on so many aspects coming together that they need to follow very specific rules and are rarely arbitrary or random. This of course is only true for important main characters, with a story and background.

For these cases it’s more than necessary to do a lot of additional work after gathering the output from a GAN. You often need to change perspective, morph some parts and collage others together. And then, of course, you re-iterate after talking to the designers, writers or a tech-artist. It can almost take as long as the regular approach to character design, either with sketches or a photo-bashing approach.

Another strong suit of these networks is for costume and clothing ideas. Here I see the benefit of being able to start with something completely weird but cool and toning it down in the process so it becomes readable and sensible for the purpose.

Another great opportunity are background characters and aliens. They are often freed from strict rules or even benefit from being unrecognizable.

Portraits

I struggled a lot with this one. The results are almost too good at times. And I was scared and disturbed after a human face that I’d just created smiled back at me with the confidence of being as real as the spilled coffee on my desk. The results are absolutely production ready. I have no doubt that in a couple of years a lot of released concept art for realistic games like The Last Of Us will drop on Artstation and nobody will ask or be surprised how these images were created so realistically.

Even more stylized or abstract faces are an easy task for a well-trained network and allow artists to quickly see how a specific character would look in a different style. Or how a character would look with more beard, or gentler features, or if their face were twice as wide, or with a different hairstyle.

In this realistic example I used a StyleGAN for the face, as it allowed me to quickly create a ¾ view of the face, show the character smiling, or to create alternative looks for the person. For the rest it was faster and more crisp to rely on traditional photobashing and overpainting it.

Environments

Next to portrait creation, environments are where the GAN’s really shine. Whether you use a mixer breeder such as Artbreeder or a segmentation map based approach such as the NVIDIA GauGAN, the results are phenomenal and really free up the creativity and speed when it comes to thumbnailing mood sketches and such.