Research Interests

Overview

My research interests lie in the interdisciplinary research that blends art and technology. Specifically, I am interested in exploring AI and HCI applications in film, theater, media, and performance. Storytelling is at the core of my interests, and I am enthusiastic to utilize AI to enhance creative expression. My goal is to utilize technologies to enable the design of immersive & interactive experiences and foster user-centered approaches.

To be more specific, I aim to:

1. Explore and develop tools/user interfaces/experimental environments for artists, filmmakers, and theater creators to facilitate their creative processes more efficiently and effectively, enabling them to produce finer and more sophisticated works.
2. Create interactive and immersive experiences by leveraging AI to promote storytelling in film, theater, and other art forms, crafting compelling narratives and enhancing audience engagement.

AI-assisted Filmmaking

One of my research interests is: AI-assisted filmmaking, which focuses on enhancing traditional filmmaking workflows by leveraging AI technologies. This includes developing tools for the pre-production, mid-production, and post-production stages to make the filmmaking process more efficient and effective. These tools will enhance pre-production tasks such as scriptwriting and storyboard creation, mid-production aspects like virtual cinematography, and post-production processes, including intelligent editing. I am enthusiastic about creating a comprehensive suite of AI-driven tools that can enhance the entire filmmaking process.

VR Theatre

I am also interested in exploring VR Theatre, where I would like to explore how XR technologies can enhance stage performance art through immersive and interactive experiences. My goal is to develop an Immersive Stage Performance Art Interaction System that allows audiences to actively engage with performances rather than passively watch them.

Currently, VR Theatre exists in two primary forms: 1. 3D-modeled metaverse environments, where scenes, characters, and narratives are digitally generated and interactive, much like a 3D game. 2. 360-degree panoramic videos, which offer a fixed set of viewpoints but lack interactivity. My goal is to integrate both approaches: using video-captured real actors and live-action scenes while enabling freeform interactivity in VR. Users can independently choose their position and role within the play. They can navigate the stage from a third-person perspective, exploring various corners of the performance space, observing characters’ expressions and emotions from different angles, appreciating the actors’ performances, or closely examining the set design and props for intricate details. Alternatively, they can immerse themselves in the perspective of a specific character, experiencing what the character sees and hears, and even understanding their thoughts and feelings, thus achieving a truly immersive theatrical experience.

To achieve this, I propose exploring Volumetric Video and 4D Gaussian Splatting to model real actors and live-action theatrical scenes. I would like to seek methods to enhance rendering speed and storage efficiency, making it feasible to reconstruct an entire theatrical performance with real actors in VR. Additionally, Generative AI, Spatial Editing, and Spatial Montage will be employed to assist in generating novel, dynamic views, enabling seamless and effective perspective synthesis.

This research has broad applications across various fields. In entertainment, it enhances theatrical experiences by merging XR with live performances, allowing audiences to engage with stage art in novel ways. In education, it serves as an interactive tool for script reading, stage blocking, and performance analysis, providing learners with a deeper understanding of theatrical production. In therapy, immersive storytelling can support psychological well-being, helping individuals process emotions and social interactions in a controlled environment.