The emergence of Generative AI systems has dramatically transformed design workflows across multiple domains. While text prompting remains the dominant interaction method with these systems, visual prompting—the practice of guiding AI generation through visual or semantic structure—offers designers potentially greater control and expressivity. A critical challenge for design practitioners lies in understanding how different visual prompting modalities integrate into established creative processes and impact design outcomes. This study examines this challenge in the context of UI mockup generation, a domain with well-defined semantic elements and established design methodologies. We evaluate two visual prompting approaches: free-form visual prompting, which creates outputs based on hand-drawn sketches, and semantic-constrained prompting, which uses predefined visual vocabularies to guide generation. Through experiments with 13 design practitioners, we explore how different prompting modalities impact both designer experience and output quality in the creation of UI artifacts. Results reveal that free-form visual prompting offers superior intuitiveness and expressiveness for ideation, while semantic-constrained prompting produces higher quality and fidelity outputs. Our findings suggest that effective visual prompting strategies should adapt to different stages of the design process, with implications for generative AI applications in design practice. We propose a hybrid approach that leverages the strengths of both modalities throughout the creative workflow, potentially offering design practitioners across domains a more balanced framework for designer-AI collaboration.
Evaluating Visual Prompting Modalities for Generative AI-Assisted UI Design / Calo, Tommaso; De Russis, Luigi. - ELETTRONICO. - (In corso di stampa). (Intervento presentato al convegno IS-EUD: the 10th International Symposium on End-User Development tenutosi a Munich (Germany) nel 16-18 June 2025).
Evaluating Visual Prompting Modalities for Generative AI-Assisted UI Design
Calo,Tommaso;De Russis, Luigi
In corso di stampa
Abstract
The emergence of Generative AI systems has dramatically transformed design workflows across multiple domains. While text prompting remains the dominant interaction method with these systems, visual prompting—the practice of guiding AI generation through visual or semantic structure—offers designers potentially greater control and expressivity. A critical challenge for design practitioners lies in understanding how different visual prompting modalities integrate into established creative processes and impact design outcomes. This study examines this challenge in the context of UI mockup generation, a domain with well-defined semantic elements and established design methodologies. We evaluate two visual prompting approaches: free-form visual prompting, which creates outputs based on hand-drawn sketches, and semantic-constrained prompting, which uses predefined visual vocabularies to guide generation. Through experiments with 13 design practitioners, we explore how different prompting modalities impact both designer experience and output quality in the creation of UI artifacts. Results reveal that free-form visual prompting offers superior intuitiveness and expressiveness for ideation, while semantic-constrained prompting produces higher quality and fidelity outputs. Our findings suggest that effective visual prompting strategies should adapt to different stages of the design process, with implications for generative AI applications in design practice. We propose a hybrid approach that leverages the strengths of both modalities throughout the creative workflow, potentially offering design practitioners across domains a more balanced framework for designer-AI collaboration.File | Dimensione | Formato | |
---|---|---|---|
paper_022 (1).pdf
accesso riservato
Tipologia:
2. Post-print / Author's Accepted Manuscript
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
802.95 kB
Formato
Adobe PDF
|
802.95 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/3000511