PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection
* Equal contribution
PRISM introduces a data-free, task-agnostic debiasing framework for VLMs. It first leverages an LLM to generate bias-aware scene descriptions from simple class prompts, then learns a linear projection of the CLIP embedding space via a novel Latent-space Debiasing loss that enforces intra-class invariance and inter-class separability.