LiquidBook Development

How Line Art Controls the Drawing: An Introduction to ControlNet’s LineArt Model

Dive into the world of ControlNet, a unique derivative of the Stable Diffusion model that has been revolutionizing image generation by providing enhanced control through the integration of extra conditions. A prominent feature of ControlNet is its LineArt model, which has been specifically conditioned to work with lineart images. This sophisticated neural network architecture is a powerful companion to Stable Diffusion implementations, such as runwayml’s stable-diffusion-v1-5.

What sets the LineArt model apart is its special knack for creating images populated with straight lines, making it an invaluable tool for various applications, from interior design visualizations to complex architectural projects. The inner workings of ControlNet are quite intriguing; it employs trainable network modules attached to the U-Net (noise predictor) of the Stable Diffusion Model. This arrangement allows the Stable Diffusion model’s weight to remain constant during training, with only the attached modules undergoing modification.

ControlNet’s adaptability doesn’t stop there. This model can be trained on a variety of input types, including but not limited to edge detection, human pose detection, and semantic segmentation. ControlNet stands as a testament to the evolution of AI in art generation, offering precision and versatility that artists and creators can harness in their work.

Line art anime

This preprocessor is an essential part of ControlNet’s LineArt model. It focuses on rendering the outlines of images in an anime style, often associated with clean and simplistic lines. This style can capture the distinctive features of subjects and is particularly suited for artistic renderings or illustrations with an anime aesthetic.

Line art anime denoise

The ‘anime denoise’ preprocessor is a variant of the anime preprocessor. While it also provides anime-style lines, it’s designed to reduce details, producing a more simplistic and abstract image representation. This approach can emphasize the primary elements of the image and remove unnecessary or distracting details, making it suitable for minimalistic anime-style artwork.

watercolor painting of a cat

Line art realistic

The ‘realistic’ preprocessor aims to create lines that mimic the complexity and intricacy found in real-world scenes. It tries to capture more detail and realism compared to the anime preprocessors. This can be especially useful for creating detailed sketches or line art drawings closely resembling the original image.

watercolor painting of a cat

Line art coarse

The ‘coarse’ preprocessor produces realistic-style lines but with heavier weight. This means it creates thicker or bolder lines, which can give the image a more dramatic or impactful look. This style can effectively emphasize certain features or elements within the image and lends itself well to striking, high-contrast artwork.

watercolor painting of a cat
Lineart coarse

The setup for generating the cat with the lineart model

Lineart preprocessor allows you the ability to control the base composition so that you can apply a style to the image through your prompt. ControlNet takes the image source and preprocesses it into a black-and-white image outline. It then takes the outline and uses it with the ControlNet model to guide the composition of the image output.

Here is one such workflow. I took a simple lineart illustration of a cat, and then in ControlNet, I selected the preprocessor Invert (from white bg & black line). With the image added to the ControlNet tab, select enable, pixel perfect (produces a more accurate outline), and allow preview checkbox options. If you click the 馃挜 icon button, ControlNet will use the preprocessor on the Image and add it to the preview panel. With the preview generated, you can drag this new Lineart Image and replace the source image. Then select none in the preprocessor dropdown menu, and the ControlNet model is set to lineart_anime.

controlnet cat invert

Then, I used the new simple inverted image with a Lineart model and generated several images until I got one I wanted to work with. Then I take the new reference image and use the lineart realistic preprocessor.

Then I take the processed image (lineart realistic) and move it from the preview to the source image. I deselect the preprocessor model since the image has already been processed. I only need it for image guidance. So with the preprocessor deselect and the Model (controller lineart) selected.

With the control values set, I add the setting for generating the image of a cat.

Prompt: a watercolor painting of a orange cat sitting on a table, painting of a cat, a painting of a cat, cat portrait painting, watercolor-wash, sunny kitchen window in the background, vibrant colors, rich orange and whites
Negative prompt: (worst quality:0.8), verybadimagenegative_v1.3, easynegative, (surreal:0.8), (modernism:0.8), (art deco:0.8), (art nouveau:0.8), extra paws, extra toes

Steps: 35, Sampler: DPM++ 2M Karras_dynthres15, CFG scale: 6.5, Seed: 2582070524, Size: 512×768, Model hash: 073447953e, Model: rundiffusionFX25D_v10, VAE: vae-ft-mse-840000-ema-pruned, Denoising strength: 0.45, Clip skip: 1, Version: 0d101b9, Parser: Full parser

Various backgrounds with the same cat in the foreground

Images were generated with the same cat but with different backgrounds.

a cat - sunny window in the background
a cat rose garden in the background
a cat - notre dame in the background
a cat - London train station in the background

I first used the illustration of a cat in a guidance test for image control in MidJourney. See the prompt post: Generating an image of a watercolor painting from a simple sketch and a text prompt.

Lean tower of Piza – Generated with Lineart

Prompt: a painting of the leaning tower of Pisa, Piazza del Duomo, by Dirck van der Lisse, inspired by Johann Balthasar Bullinger, golden hour, blue sky above
Negative prompt: (worst quality:0.8), verybadimagenegative_v1.3, easynegative, (surreal:0.8), (modernism:0.8), (art deco:0.8), (art nouveau:0.8), style-empire-neg
Steps: 30 | Sampler: DPM++ 2S a Karras | CFG scale: 7 | Seed: 1224046228 | Size: 408×608 | Model hash: 073447953e | Model: rundiffusionFX25D_v10 | VAE: vae-ft-mse-840000-ema-pruned | Denoising strength: 0.7 | Clip skip: 1 | Version: b48a0f1 | Token merging ratio: 0.5 | Token merging ratio hr: 0.5 | Parser: Full parser | ControlNet 0: “preprocessor: none | model: control_v11p_sd15s2_lineart_anime [3825e83e] | weight: 1 | starting/ending: (0 | 1) | resize mode: Crop and Resize | pixel perfect: True | control mode: ControlNet is more important | preprocessor params: (512 | 64 | 64)” | Hires upscale: 2 | Hires upscaler: R-ESRGAN 4x+ Anime6B

Note: The lineart image was modified using Affinity Photo, and the lines for the background were removed.

Superman Art Generated with ControlNet Lineart

superman man of steel - screen clip - used for reference
superman lineart
Superman AI Generated Art from lineart

Prompt: a close up of a superman, moody iconic scene, pleading face, standing in a barren field, dramatic sky in the background, dawn morning, sublime-cool-hot-hyperadvanced, pained expression, by Jim Lee, perfect symmetric coherent face, a broad shouldered, solemn, short hair cut,
Negative prompt: (worst quality:0.8), verybadimagenegative_v1.3, easynegative, (surreal:0.8), (modernism:0.8), (art deco:0.8), (art nouveau:0.8), hair over face, hair cover face
Steps: 30 | Sampler: DPM++ 2M Karras | CFG scale: 7 | Seed: 3684907572 | Size: 664×664 | Model hash: 073447953e | Model: rundiffusionFX25D_v10 | VAE: vae-ft-mse-840000-ema-pruned | Denoising strength: 0.4 | Clip skip: 2 | Version: b48a0f1 | Token merging ratio: 0.5 | Token merging ratio hr: 0.5 | Parser: Full parser | ControlNet 0: “preprocessor: none | model: control_v11p_sd15s2_lineart_anime [3825e83e] | weight: 1 | starting/ending: (0 | 1) | resize mode: Crop and Resize | pixel perfect: True | control mode: Balanced | preprocessor params: (512 | 64 | 64)” | Hires upscale: 1.5 | Hires upscaler: R-ESRGAN 4x+ Anime6B

Note: The lineart image was modified using Affinity Photo, and the lines for the background were removed. The hairline was corrected, and some added lines were removed from the face.

Checkpoint Model: RunDiffusion FX 2.5D
Website: RunDiffusion FX 2.5D

Social Sharing
Design Wireframe
website wireframe for a web app, blue and orange. 3D, white background, photorealistic, minimal style
DALL路E-2024-04-04-10.01.32-A-wide-format-HD-quality-image-showcasing-a-beautiful-set-of-cottages-in-the-Scottish-coastal-town-of-Tobermory-rendered-with-extreme-detail-and-pho
image: a beautiful set of cottages, Scotland coastal town of Tobermory, extremely detailed, photorealistic, cinematic...
liquidbook_8k_photo_realistic_colorful_bright_light_day_time_ou_1febbd3d-5bcb-43b0-a759-e6135be8ed7d
8k, photo, realistic, colorful, bright light, day time, outdoor, Gundam GP04, extreme detailed armor, giant heavy wea...
liquidbook_concept_art_of_a_lean_horse_the_horse_is_white_and_h_0a3ff1b8-378c-46c2-b1d4-0131421519f0
concept art of a lean horse, the horse is white and his legs are black, big brush strokes, static position

This website stores cookie on your computer. These cookies are used to collect information about how you interact with our website and allow us to remember you. We use this information to improve and customize your browsing experience and for analytics and metrics about our visitors both on this website and other media. To find out more about the cookies we use, see our Privacy Policy.