How to Use ControlNet Depth Maps to Fix AI Composition Problems

When Prompting Can't Fix the Problem

There are composition problems in AI art that prompting cannot solve. The foreground element always ends up centered when you need it left. The horizon keeps landing at the wrong height. Background elements crowd the subject no matter how many descriptors you add. Depth relationships between layers are flat when you need clear separation.

These are structural problems. Text prompts influence what content appears and in what style, but they have limited control over precise spatial arrangement. When you need a specific compositional structure, ControlNet with a depth map is the right tool.

What ControlNet Actually Does

ControlNet is a neural network extension for Stable Diffusion (and SDXL) that conditions the generation on an additional input image, rather than just text. Instead of the model making all spatial decisions from the prompt alone, ControlNet provides a structural reference that constrains the spatial arrangement of the output.

There are several ControlNet types: pose (for figure position), canny edge (for hard structure preservation), lineart, and depth, among others. Depth maps are specifically useful for controlling the three-dimensional layering and spatial arrangement of a scene, which makes them the most relevant type for compositional control of environments, cityscapes, and atmospheric scenes.

What a Depth Map Is

A depth map is a grayscale image where brightness represents distance from the camera. Lighter areas are closer to the viewer. Darker areas are further away. This encoding gives the model a spatial instruction that describes not just where elements are in the 2D frame but how they relate in three-dimensional space.

When you feed a depth map into ControlNet, the model uses it as a structural blueprint. Generated content that is spatially close to the camera will appear in the lighter zones. Background content will appear in the darker zones. The layering, foreground-to-background separation, and depth relationships in the output will broadly follow the depth map.

How to Create a Depth Map for Composition Control

You have three practical options for creating a depth map:

Option 1: Extract a depth map from an existing image. Use the ControlNet preprocessor (MiDaS or ZoeDepth) built into ComfyUI or Automatic1111 to analyze any image and extract its depth structure. This is useful when you have a reference composition you like and want to use its spatial structure while generating completely different content and style.

Option 2: Paint a depth map manually. Open a canvas (Photoshop, Affinity Photo, GIMP, or even a basic paint tool) at the same resolution as your target output. Paint in grayscale: white where you want foreground elements, progressively darker grays for mid and background layers, black for the most distant background. This is faster than it sounds for simple compositional structures and gives you precise control over exactly what you want.

Option 3: Use a 3D scene or reference render. More advanced workflows use simple 3D scenes rendered in Blender or other tools as depth reference sources. This is overkill for most use cases but produces precise spatial control for complex architectural or environmental compositions.

Setting Up a Depth ControlNet Workflow in ComfyUI

The basic workflow structure:

1. Load your base model (SDXL or SD 1.5, matching your ControlNet model)
2. Add a ControlNet loader node and select the depth-compatible ControlNet model (depth_fp16 or depth-midas for SD1.5; controlnet-depth for SDXL)
3. Connect your depth map image as the control image input
4. Set the ControlNet conditioning strength. Start between 0.6 and 0.8. Higher values follow the depth map more closely but may constrain the model too tightly. Lower values give the model more freedom but reduce the compositional constraint.
5. Connect the ControlNet output into your standard KSampler via the positive conditioning input
6. Write your text prompt describing the content and style you want layered onto the depth structure

The depth map tells the model where things are. The text prompt tells the model what they are and what they look like. Used together, you get much more compositionally controlled outputs than either can produce alone.

Practical Use Cases for Wallpaper Generation

Controlling foreground-background separation. A common problem in AI cityscape generation is a flat, undifferentiated depth field where foreground and background elements compete visually. A depth map with strong white foreground elements, a mid-gray street layer, and dark background buildings forces the model to create genuine depth separation rather than a flat composite.

Fixing horizon line position. AI models tend to place the horizon in compositionally median positions. If you need a low horizon for an open sky composition or a high horizon for a worm's-eye view, painting this into a depth map constrains the spatial arrangement reliably.

Controlling element placement in scenes with multiple layers. For compositions where you need a specific structural arrangement (subject in left foreground, atmospheric mid-ground, deep architectural background), painting this spatial structure into a depth map is more reliable than prompting for it.

ControlNet Strength: How to Calibrate It

The strength parameter controls how closely the output follows the depth map versus how freely it interprets the text prompt.

At 0.3 to 0.5: the depth map provides loose guidance. The model follows the general spatial structure but has significant freedom to reinterpret it.

At 0.6 to 0.8: strong compositional constraint. Spatial arrangement closely follows the depth map. This is the practical working range for most composition control use cases.

At 0.9 to 1.0: very tight constraint. The output follows the depth map rigidly. This can produce stiff or overly literal results, especially with hand-painted depth maps that have sharp transitions.

If your outputs look spatially correct but visually stiff, lower the strength slightly. If the composition is drifting from your depth map intent, raise it.

What Depth Maps Can and Cannot Control

Depth maps control spatial arrangement and depth layering. They do not:

• Control specific object identity or subject matter (use text prompts for this)
• Control style, color, or lighting (use text prompts and style references for these)
• Guarantee exact element placement — there is always some interpretive latitude, especially at lower strength values
• Work identically across different base models — the same depth map produces different spatial interpretations with different models

Common Mistakes

Using a depth ControlNet model that doesn't match the base model version. SD 1.5 ControlNet models do not work with SDXL and vice versa. Always match the ControlNet model to your base model.

Painting the depth map at the wrong resolution. The depth map should be the same resolution as your target output. A mismatch causes spatial scaling artifacts.

Setting strength too high on a rough hand-painted depth map. Sharp transitions in a hand-painted map at high strength produce visible seams in the output. Blur the depth map slightly before use to soften transitions.

Expecting exact element reproduction from a depth map. The model uses the depth map as a spatial constraint, not a pixel-level blueprint. The general structure follows; the specific content does not.

Using depth ControlNet when a different type would solve the problem better. Depth is for spatial arrangement. If the problem is a figure in the wrong pose, use pose ControlNet. If the problem is specific architectural shapes, use canny edge ControlNet.

Composition Control Checklist

• Identified the specific compositional problem that prompting cannot solve
• Selected the correct ControlNet type for the problem (depth for spatial arrangement)
• Depth map is at the same resolution as the target output
• ControlNet model matches the base model version (SD1.5 or SDXL)
• Starting strength set to 0.6 to 0.8 and adjusted based on test outputs
• Text prompt describes content and style; depth map handles spatial structure
• Depth map transitions are soft enough to avoid seam artifacts at the chosen strength
• Tested at multiple strength values before settling on a final value