Evaluating Visual Prompting Modalities for Generative AI-Assisted UI Design

The emergence of Generative AI systems has dramatically transformed design workflows across multiple domains. While text prompting remains the dominant interaction method with these systems, visual prompting—the practice of guiding AI generation through visual or semantic structure—offers designers potentially greater control and expressivity. A critical challenge for design practitioners lies in understanding how different visual prompting modalities integrate into established creative processes and impact design outcomes. This study examines this challenge in the context of UI mockup generation, a domain with well-defined semantic elements and established design methodologies. We evaluate two visual prompting approaches: free-form visual prompting, which creates outputs based on hand-drawn sketches, and semantic-constrained prompting, which uses predefined visual vocabularies to guide generation. Through experiments with 13 design practitioners, we explore how different prompting modalities impact both designer experience and output quality in the creation of UI artifacts. Results reveal that free-form visual prompting offers superior intuitiveness and expressiveness for ideation, while semantic-constrained prompting produces higher quality and fidelity outputs. Our findings suggest that effective visual prompting strategies should adapt to different stages of the design process, with implications for generative AI applications in design practice. We propose a hybrid approach that leverages the strengths of both modalities throughout the creative workflow, potentially offering design practitioners across domains a more balanced framework for designer-AI collaboration.