--- title: SILHOUETTE Studio emoji: 🔲 colorFrom: gray colorTo: gray sdk: gradio sdk_version: "4.42.0" python_version: "3.10" app_file: app.py pinned: false --- # SILHOUETTE Studio > **Minimalist Neural Pose Synthesis & Rapid Latent Consistency Engine** > Built with **PyTorch**, **HuggingFace Diffusers**, **ControlNet (OpenPose)**, and **LCM-LoRA** on **Stable Diffusion 1.5**. --- ## 1. Executive Summary **SILHOUETTE Studio** is a high-performance generative platform designed for real-time spatial pose conditioning and text-to-image synthesis. Traditional diffusion pipelines require 30 to 50 denoising iterations per image, taking anywhere from 10 to 30 seconds on consumer GPUs. **SILHOUETTE Studio** fuses **Latent Consistency Models (LCM-LoRA)** directly into the base UNet's cross-attention layers, allowing the pipeline to predict clean latent destinations in **4 to 8 steps (~1.5 to 3.0 seconds)** without compromising anatomical fidelity or structural pose alignment. --- ## 2. Core Capabilities | Feature | Technical Backbone | Description | |---|---|---| | **Pose Studio** | ControlNet OpenPose (`control_v11p_sd15_openpose`) | Extracts skeletal joint coordinates from reference photographs or accepts hand-drawn poses directly from the canvas to enforce strict anatomical posture onto the output image. | | **Fast Synthesis** | SD 1.5 + LCM-LoRA (`lcm-lora-sdv1-5`) | Rapid text-to-image latent consistency generation. Solves the generative trajectory in 4 steps with Classifier-Free Guidance set to 1.0. | | **High-Fidelity VAE** | StabilityAI `sd-vae-ft-mse` | Fine-tuned Mean Squared Error autoencoder that eliminates color banding, artifacting, and skin-tone clipping common in the original SD 1.5 VAE. | --- ## 3. High-Level Architecture & Data Flow ``` [ User Input ] ──┬──► Text Prompt ──────────► [ CLIP Text Encoder ] ─────────► (1, 77, 768) Embedding │ ├──► Reference Photo ──────► [ OpenPose Detector ] ──────┐ │ │ └──► Canvas Sketch ──────────────────────────────────────┴──► (1, 3, 512, 512) Pose Tensor │ ▼ [ Gaussian Noise Tensor ] ◄────────────────────────────────────────────────── [ ControlNet v1.1 ] z_T: (1, 4, 64, 64) │ │ │ Extracts 13 multi-scale ▼ │ residual feature maps ┌────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┐ │ Iterative LCM Denoising Loop (4–8 Steps) │ │ │ │ z_t ──► [ UNet + Fused LCM-LoRA ] ◄── (ControlNet Residuals * CN_Scale) ──► Predicts Clean Latent z_0 │ │ │ │ │ └──► [ LCMScheduler ] ──► Calculates Step z_(t-1) ──► (Repeats for N steps) │ └────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┘ │ ▼ Final Clean Latent z_0 (1, 4, 64, 64) │ ▼ [ VAE Decoder (ft-mse) ] │ ▼ Output PIL RGB Image (1, 3, 512, 512) ``` --- ## 4. Deep-Dive: How the Models Work Together ### A. The Difference Between Standard Diffusion and LCM * **Standard Diffusion (Velocity Prediction)**: Standard SD models solve a curved Ordinary Differential Equation (ODE) trajectory by taking 30–50 tiny incremental baby steps along the noise curve. If you try to jump 10 steps at once, the calculation veers off the manifold into distorted garbage. * **Latent Consistency Model (Destination Prediction)**: LCM is parameterized with a consistency function $f(z_t, t) \to z_0$. Regardless of the current noise level $t$, the UNet is trained to predict the **final clean image $z_0$ directly in a single leap**. Running 4 steps simply repeats this prediction from successively clearer vantage points, refining fine details (eyes, skin texture, reflections). ### B. Why Low Step Counts (4–8) are Mandatory with LCM * **4–6 Steps (Sweet Spot)**: Optimal balance of contrast, sharpness, and prompt adherence. * **20+ Steps (Degradation)**: Because LCM was distilled specifically for sparse leaps, running excessive steps causes small mathematical approximation errors ($~0.5\%$ per step) to compound and multiply, resulting in burnt contrast, oversaturation, and noise artifacts. ### C. ControlNet Residual Injection ControlNet creates a locked copy of the UNet's 12 encoder blocks and 1 middle block. It takes the $512 \times 512$ OpenPose skeleton image, encodes it through zero-convolution layers, and outputs 13 residual feature maps. During each step of the denoising loop: $$\text{UNet\_Layer\_Input} = \text{UNet\_Features} + (\alpha \cdot \text{ControlNet\_Residual})$$ where $\alpha$ is the `ControlNet Conditioning Scale` (default `0.85`). This guarantees that body posture, arm angles, and limb positions are strictly adhered to while allowing the base model full freedom to generate clothing, backgrounds, and artistic style. --- ## 5. Model Inventory | Subsystem | Model ID | VRAM (fp16) | Purpose | |---|---|---|---| | **Base UNet** | `runwayml/stable-diffusion-v1-5` | ~1.7 GB | Core latent denoising network | | **ControlNet** | `lllyasviel/control_v11p_sd15_openpose` | ~1.4 GB | Spatial skeletal pose conditioning | | **Fast Sampler** | `latent-consistency/lcm-lora-sdv1-5` | ~0.1 GB (fused) | Consistency trajectory acceleration | | **Autoencoder** | `stabilityai/sd-vae-ft-mse` | ~0.3 GB | $64 \times 64 \leftrightarrow 512 \times 512$ latent/pixel mapping | | **Text Encoder** | `openai/clip-vit-large-patch14` | ~0.6 GB | Text prompt semantic embedding | | **Pose Detector** | `lllyasviel/ControlNet` (via `controlnet_aux`) | ~0.3 GB | Computer vision skeleton keypoint extractor | | **Total VRAM** | **Combined Peak Pipeline** | **~4.8 GB - 5.5 GB** | Fits easily on RTX 3060/4060, T4, or A10G | --- ## 6. Deployment Guide: Where and How to Host Because diffusion models require ~6 GB of weights and a GPU with at least 6 GB VRAM, traditional serverless providers (Vercel, AWS Lambda, Heroku) cannot host this directly. Below are the **top 3 recommended deployment options**: --- ### Option A: Hugging Face Spaces (Recommended - Direct Git Push) Hugging Face Spaces natively recognizes the YAML metadata at the top of this `README.md`. 1. Create a free account at [huggingface.co](https://huggingface.co). 2. Click **New Space** -> Name it `silhouette-studio` -> Select **SDK: Gradio**. 3. Choose **ZeroGPU** (free dynamic A100 GPU) or **T4 Small ($0.60/hr)**. 4. Push this repository to your Space: ```bash git remote add space https://huggingface.co/spaces//silhouette-studio git push space main ``` 5. Hugging Face will parse the frontmatter, install `requirements.txt`, and automatically boot `app.py`. --- ### Option B: Modal Labs (Serverless GPU - Cost Efficient) [Modal](https://modal.com) allows you to run PyTorch containers on serverless NVIDIA A10G/T4 GPUs with zero idle server cost. **Example `modal_app.py` wrapper:** ```python import modal app = modal.App("silhouette-studio") image = ( modal.Image.debian_slim(python_version="3.10") .pip_install_from_requirements("requirements.txt") .pip_install("torch", "torchvision", index_url="https://download.pytorch.org/whl/cu121") ) @app.function(gpu="T4", image=image, container_idle_timeout=60) @modal.web_server(port=7860) def serve(): from app import build_app demo = build_app() demo.launch(server_name="0.0.0.0", server_port=7860) ``` Deploy with: ```bash modal deploy modal_app.py ``` --- ### Option C: RunPod / Vast.ai (Dedicated GPU - High Traffic) If you need a dedicated instance running 24/7 for ~$0.20/hour: 1. Deploy a **RunPod Secure Cloud instance** with an **RTX 4000 Ada (20GB)** or **T4 (16GB)**. 2. Select the `runpod/pytorch:2.1.0-py3.10-cuda12.1.1-devel-ubuntu22.04` template. 3. SSH into the pod: ```bash git clone https://github.com/your-username/real-aura.git cd real-aura pip install -r requirements.txt python app.py ``` 4. Connect via the provided HTTP proxy port `7860`. --- ## 7. Local Setup & Installation ### Prerequisites * **Operating System**: Windows 10/11, Linux, or macOS (Apple Silicon). * **GPU**: NVIDIA GPU with $\ge 6 \text{ GB VRAM}$ recommended. * **System RAM**: $\ge 16 \text{ GB}$. * **Disk Space**: $\approx 8 \text{ GB}$ free for model cache. ### Step 1: Clone the Repository ```bash git clone https://github.com/your-username/real-aura.git cd real-aura ``` ### Step 2: Install PyTorch with CUDA Install PyTorch compiled with your matching CUDA version (example for CUDA 12.1): ```bash pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121 ``` ### Step 3: Install Project Dependencies ```bash pip install -r requirements.txt ``` ### Step 4: Launch the Studio ```bash python app.py ``` The application will launch and automatically open in your default browser at: ``` http://localhost:7860 ``` > **Note on Initial Startup:** On the very first generation, Diffusers downloads the base weights (~6 GB total) from Hugging Face Hub to `./models/`. Subsequent boots load directly from cache in ~10 seconds. --- ## 8. Parameter Guide & Recommended Settings | Parameter | Recommended Value | Description | |---|---|---| | **Steps (Pose Studio)** | `6 – 8` | Number of consistency denoising steps. Allows UNet to harmonize the pose with prompt semantics. | | **Steps (Fast Synthesis)** | `4` | Pure text-to-image optimal consistency step count. | | **CFG Scale** | `1.0 – 1.5` | Classifier-Free Guidance scale. LCM requires low CFG. Values above 2.0 cause contrast blowouts. | | **ControlNet Scale** | `0.75 – 0.90` | Strictness of pose conformity. `0.85` matches limbs accurately while preserving natural proportions. | | **Seed Mode** | `Random` / `Fixed` | Use `Fixed` with an integer (e.g. `42`) to reproduce exact composition and lighting variations. | --- ## 9. Project Directory Structure ``` real-aura/ ├── app.py # Gradio web application & monochrome UI styling ├── pipeline/ │ ├── __init__.py # Package exports │ ├── loader.py # Singleton model loader (VAE, UNet, ControlNet, LCM-LoRA) │ ├── processor.py # OpenPose detection, sketch parsing & normalization │ └── generate.py # Core inference functions (generate_with_pose, generate_t2i) ├── utils/ │ ├── __init__.py # Package exports │ ├── canvas.py # Skeleton canvas overlay & OpenPose joint color definitions │ └── image_utils.py # PIL <-> NumPy conversions, aspect ratio resizing & output saving ├── models/ # Model cache directory (auto-created, gitignored) ├── outputs/ # Image archive directory (auto-created, gitignored) ├── requirements.txt # Production dependency specifications └── README.md # Technical architecture & deployment documentation ``` --- ## 10. License This project is distributed under the Apache-2.0 License. Base model weights (`stable-diffusion-v1-5`, `controlnet_openpose`, `lcm-lora`) are subject to their respective OpenRAIL and CreativeML licenses.