Type: Individual project | Independent study | Timeline: 2023 (7 months) | Role: UX designer & Software engineer
Supporting industrial design students to better visualise and validate their 3D models in realistic scales and environments with XR and generative AI.
Scroll ↓
How might we leverage virtual reality to assist industrial design students in visualising and testing their 3D models in realistic scales and environments?
Brief
Is virtual reality technology just hype?
Is it merely excitement over a new technology because of the imagined future it may one day provide, in contrast to what is realisable? Or is there something more profound about it that has been overlooked — could it contribute to solving real problems, given its current state?
With these questions in mind, I set out to do an independent study, with support from two professors at UPenn, on how virtual reality may bring value to something tangible in my current environment — specifically towards improving the learning outcomes for industrial design students today.
Outcome
A fully-functioning virtual and augmented reality platform for industrial designers to visualise and validate their 3D models in realistic scale and contexts using generative AI — www.synexr.com.
Posted demos that went viral on Twitter with over 40,000 impressions. Early testers and advisors included directors and designers at Autodesk, MIT Media Lab, IDEO, Microsoft Research, Mancini Duffy, and Facebook Reality Labs.
Final demo
I interviewed multiple professors at industrial design schools to identify key challenges of teaching industrial design.
I interviewed five programme directors and professors from Pratt Institute, MIT, Carnegie Mellon, and the University of Pennsylvania.
Key quotes from interviewed stakeholders
FigJam workspace for mapping research & ideation
Key takeaways from interviews
Industrial design students share important struggles in their design process. They grapple with the tendency to converge on ideas too quickly, spend excessive time building realistic models for user testing, struggle with conducting thorough and successful research, and face difficulties in obtaining timely feedback from real users in relevant contexts.
The significance of the challenges and how feasible it may be to solve them vary.
Therefore, to prioritise my efforts, I examined the viability and feasibility of solving the discovered challenges, and focused on three specific pain points.
01
The time-consuming process of building realistic models for testing
02
The reduced access to relevant environments to test ideas
03
The lack of feedback from real users in relevant contexts
By combining the prioritised pain points, I formulated one overarching problem statement:
How might we leverage virtual reality to assist industrial design students in visualising and testing their 3D models in realistic scales and environments?
This is a key problem to address for all industrial designers, irrespective of their student status. Being capable of better visualising and testing one’s 3D models will allow designers to work more efficiently and save significant costs by:
01 Discovering critical design mistakes earlier
02 Making more informed design iterations
03 Accessing realistic environments for early testing
04 Working more time and cost-efficiently with materials, 3D printing, and painting
Outlined target persona based on key insights from the stakeholder interviews
I brainstormed various potential solutions — from design critiques to real-time modelling in VR.
I then audited my ideas based on their potential impact on solving the defined problem and the effort required to implement them. I also prioritized ideas that would stretch my learning and take advantage of this opportunity to enhance my technical skills.
Key ideas audited based on the potential impact to solve the defined problem and the effort required to implement them
I analysed four competitive solutions in the market
I further sought to understand what similar virtual reality solutions in the market lacked in addressing the challenges I was targeting and how I might offer a better and differentiated value proposition.
Spider graph showing a competitive analysis of four market incumbents.
Key takeaways from competitive analysis
Most potential competitors primarily target architects and 3D modelling in VR. They fail to cater to industrial designers, user testing, they provide limited options for visualizing models in different contexts, they lack the ability to explore models in various materials, and they have very steep learning curves.
Ultimately, most of my ideas could be combined into a cohesive design tool to offer a truly differentiated value proposition
My goal became to create a new design tool allowing designers to seamlessly upload their 3D models from their laptops into virtual reality to further explore. This tool allows them to experiment with scale, materials, lighting, virtual environments, and gravity to better visualize and validate their models. Also, by connecting the VR headset to a monitor, designers can see how others may interact with their models, which may be great for user testing.
Early demos
I conducted extensive testing with 12 different design students, hosting over 20 different user testing sessions with them throughout the school semester
The feedback varied depending on the level of refinement of the prototypes, but it could be distilled down to three key points.
01
Students found immense value in visualizing their models within realistic contexts.
02
Students valued the ability to interact with and explore their designs in ways they had never before.
03
Students recognized the tool as a valuable supplement to rendering software like Keyshot and 3D modelling tools, enhancing the visualization process during the modelling phase.
The feedback also highlighted five specific barriers that needed to be addressed for the tool to increase user adoption and retention:
01
The realism of materials and environments was not convincing enough.
02
Interactions with the models felt unnatural.
03
Users desired more control over model customization, environment adjustments, and material selection.
04
Users wished to capture their VR experiences to share with teammates.
05
Users wanted the process of uploading 3D models into VR to be more seamless.
I drew value from the recent developments in AI
As I was building my application, the world was introduced to new powerful text-to-image AI models such as DALL-E and Blockade Lab. I saw the potential of such models to address many of the user adoption barriers I discovered and explored ways to incorporate them.
Ultimately, I combined real-time voice interactions with text-to-image models to allow users to intuitively generate high-fidelity materials and entire virtual environments to better visualise and explore their 3D models.
Final key features
01 Uploading 3D models from Google Drive
Through an intuitive in-app onboarding flow, users are able to directly upload any 3D model stored in their personal Google Drive, into virtual reality.
02 Exploring models in augmented reality
Users can visualise and explore their 3D models in their current environments with full augmented reality capabilities.
03 Generating textures with AI
Using simple voice interactions, users can generate advanced high-definition textures applied to their 3D model.
04 Generating environments with AI
Users are able to use their voice to generate high-definition environments they can use to better visualise their model in realistic contexts.
05 Physics, scaling, lighting, snapshots, copying and more.
Users can explore multiple iterations of their designs with gravity, in different lighting, in various scales, save snapshots and share them with their team.
Next steps
My next steps are to build a full-stack web platform, making the 3D model uploading process into VR more seamless and supporting more advanced capabilities, such as editing distinct parts of a model.
What I learned
This project challenged my skills in designing humane and effective interfaces and interactions for a new medium, such as virtual/augmented reality and generative AI.
I got the opportunity to enhance my programming abilities and better understand the workflows and trade-offs between design and development/implementation.
I learned how to better identify and synthesise pain points end-users (students) may not be aware of, but that related external stakeholders (professors) have nuanced first-hand insights about.