Image Generation Engineering

The Control Stack: ControlNet / LoRA / IP-Adapter

Background & principle

These three tools act on different facets of the diffusion process—not substitutes, but layers:

• ControlNet (Zhang 2023): clones the UNet encoder and injects spatial conditions (canny edges, depth, openpose skeleton, segmentation) via zero-conv. Pixel-level composition/pose lock, at ~30%+ extra inference cost.
• LoRA (Hu 2021): low-rank weight deltas (rank ~4–128); fine-tunes a style or character from dozens of images, a few-MB file activated by a trigger word. The workhorse for subject consistency.
• IP-Adapter: encodes a reference image into an image prompt injected into cross-attention—zero training to transfer style/face. Faster than LoRA, but weaker control.

The engineering mantra: fix structure first (ControlNet), then identity (LoRA), then tune mood last (prompt + IP-Adapter). Route "what you want" to the right control channel instead of cramming it all into the text prompt.

Hands-on

# diffusers: ControlNet(pose) + LoRA(character) stacked
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
cn = ControlNetModel.from_pretrained("...controlnet-openpose-sdxl")
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=cn)
pipe.load_lora_weights("./my_character.safetensors")  # consistency

img = pipe(
    prompt="mychar, cinematic lighting, rooftop at dusk",
    image=pose_map,                 # openpose skeleton locks pose
    controlnet_conditioning_scale=0.8,  # don't use 1.0; leave slack
    guidance_scale=6.5, num_inference_steps=30,
    generator=torch.Generator().manual_seed(42),  # reproducible
).images[0]

Image Prompts: Completely Different Rules from LLMs

Background & principle

Bringing LLM prompting habits to diffusion is almost all wrong. Four core differences:

• Negation fails: writing "don't add a hat" makes a hat more likely because you mentioned it. Put exclusions into a separate negative_prompt channel.
• CFG scale = obedience: classifier-free guidance (Ho & Salimans 2022) extrapolates between conditional and unconditional predictions. Too low (<4) drifts off-prompt; too high (>14) oversaturates with artifacts. SDXL typically uses 5–8.
• Order is weight: earlier tokens carry more influence; most UIs also support explicit (word:1.3) weighting.
• 77-token hard cut: the CLIP text encoder takes only 77 tokens; the overflow is silently dropped—don't write essays.

Hands-on

# ❌ LLM-style (almost always backfires on diffusion)
"Please generate a portrait. Make sure there is no blur,
 and don't include any text or watermark in the image."
# → mentioning blur/text/watermark summons them; instruction filler eats 77 tokens

# ✅ Diffusion-style: positive holds only what you WANT, exclude via negative
prompt = "portrait of a woman, soft window light, 85mm, \
          shallow depth of field, (sharp focus:1.2)"
negative_prompt = "blurry, text, watermark, extra fingers, \
                   lowres, deformed hands"
# guidance_scale=7  order: subject→light→lens→quality

Image Generation Agent: Turning Gacha into a Generate–Critique–Repaint Loop

Background & principle

Production won't accept "render ten, pick one." Wire image generation into an agent loop, reusing the harness thinking you already know (Day 3): generate → VLM evaluates against the brief → if it fails, mask the problem region + inpaint a local repaint → upscale to finalize. The key is a multimodal model (Claude / GPT-4V) acting as an eval gate, automating "human picks the image" into a structured score. Repaint only fixes the bad local region (hands, text, stray objects) instead of re-rolling the whole image—saving cost and preserving the parts already correct.

Hands-on

# VLM as reviewer: structured score drives whether to repaint
import anthropic
client = anthropic.Anthropic()

def critique(image_b64, brief):
    msg = client.messages.create(
        model="claude-opus-4-8", max_tokens=400,
        messages=[{"role":"user","content":[
            {"type":"image","source":{"type":"base64",
             "media_type":"image/png","data":image_b64}},
            {"type":"text","text":
             f"Brief: {brief}\nScore strictly, return JSON: "
             '{"pass":bool,"issues":[...],"worst_region":"hands|text|..."}'}
        ]}])
    return msg.content[0].text   # → parse, decide which region to inpaint

# loop: max_iters=3, escalate to human beyond that, avoid burning $$ on infinite repaints

Watermarking & Copyright: The Provenance Layer You Can't Skip

Background & principle

Ship a generated image into a product and compliance surfaces immediately. Engineering splits into two layers:

• SynthID (Google DeepMind): embeds an invisible watermark distributed across the whole image's pixels, surviving screenshots, resizing, JPEG recompression, and minor crops—it persists even after metadata is stripped. By 2026 SynthID + C2PA has become a cross-vendor de facto standard.
• C2PA Content Credentials: attaches a cryptographically signed chain of origin (who, which model, how edited). Verifiable and traceable, but a single screenshot or re-save can strip it—so pair it with SynthID.
• Copyright gray zone: the real legal risk is not "does the output carry a watermark" but the training data and style replication. Using LoRA to clone a living artist's style, or fine-tuning on someone's work, is a high-risk zone.

Hands-on

# Sign output with C2PA Content Credentials (c2patool)
c2patool input.png \
  --manifest manifest.json \   # declare: AI-generated + model + time
  --output signed.png

# key fields in manifest.json
{
  "claim_generator": "my-pipeline/1.0",
  "assertions": [
    {"label": "c2pa.actions",
     "data": {"actions": [{"action": "c2pa.created",
              "digitalSourceType": "trainedAlgorithmicMedia"}]}}
  ]
}
# two layers: C2PA signature (strippable but verifiable) + platform SynthID (robust, invisible)