Motion Capture animation integrated into live events with Unreal Engine 5.1 represents one of the most innovative technology’s sides.
The technological developments for live events are one of the most innovative resources implemented in the latest years. The interaction with the audience and the entertainment decree the event’s success, and this is why live event organizers integrate these creative choices into their shows.
The last live event showed us the potential of these new technologies and how they can add value to events and audiences.
Motion Capture Live: how we animated live two digital avatars
The latest creative idea of Màgina, consultant to national and international groups in the design and management of communication events, and Mezcal Agency, a creative content factory of digital communication, was a fully animated show in Motion Capture, with two lead actors playing two cartoon-style characters who, projected onto the stage, interacted with the presenter in real time.
Two Motion Capture set up: Xsens suit for body movement tracking, Manus Meta gloves for hand animation, and Apple ArKit for face tracking. The two characters were then animated 360°, from body movements to the expressiveness of both faces.
The two characters, modeled and customized with Character Creator 4 by WondAR team, were later placed in a digital environment of Unreal Engine 5.1 and prepared for Xsens animation.
Like any live event, direction was also set for the two 3D protagonists. Six shots alternated during the event: close-ups, close medium shot, and whole figures.
These technologies' potential add value to live events, by allowing creative choices to be made that would not be possible without these set up.
The interaction with the presenter was an example of how this can be integrated with live events, and how there is a way to create interactive dynamics even with the audience attending such an event.
Motion Capture Live: interview with Rob Budde
Introduce yourself to our blog readers, what is your background?
My background is a long journey, starting in Holland with theater school with specialization in theater and TV directing. I'm going to tighten my professional/creative history to the maximum, which passes through experimental theater, documentaries and live programs on TV (holland), directing assistance in cinema, organization of large live events to conception and realization of experiential events for companies.
Lately I have been working with my agency Màgina on creative events, often within congresses, communication and presentation of new concepts and new product launches. My passion is the live event, directly with the audience, the guests, where the emotional experience is the key to the audience's attention and emotional involvement.
What was your first approach to motion capture?
I've always had a great passion for the creative application of visual technology, from the first experiments now decades ago, with a live camera in a theater show, to video mapping, to interactive holographic projections, to the use of cinema technology in an interactive and live key with motion capture.
Five years ago we, on the intuition of a choreographer friend of ours, began applying this technology at an annual convention of a large pharmaceutical company, using actors who could interact live with the speakers on stage through avatars, projected on a large screen. The inclusion of this emotional experience greatly stimulated the convention attendees and greatly increased the audience's level of attention and learning.
The inclusion of interactive dialogue between on-stage reality and virtual reality (avatars) allows commenting on the topic from the point of view of a third person, a (pretend) colleague in the audience, or a specialist in the field (journalist-scientist...) in a surprising, engaging, brilliant, comical way. With the inclusion of these technologies we are able to enhance the communicative power of the event.
Another application of "visual and sensorial technology" can be found as a 'teaching' tool in experiential education and professional training. The tool of AR/ VR and the use of avatar through motion capture are very useful in training meetings, where we work on skills such as awareness, interpersonal relationships, listening and responsiveness.
These technologies allow experimentation of behaviors in the safety of a simulation of reality where the participant can try different styles of management and behavior without hurting themselves (we are in a different reality...).
Do you think that the actors you choose for these types of shows must have a particular acting ability to render the idea of the character on the big screen?
Actors working with motion capture must have great improvisational skills, be able to get into character immediately and be prepared to intervene with knowledge of the subject matter. In addition, they must have good control of mimic, body and facial expression, since the range of physical action is very limited.
This is your second time collaborating with WondAR Studios, what progress have you made in the past two years?
The collaboration with WondAR Studios was one of the reasons why we continued on this path. Motion capture technology is mainly applied in video and film making.
The challenge of bringing the technology to live situations where it is not possible to process images in post production was met by WondAR with great passion and professionalism. From the first live productions, where the results were in any case very satisfactory, we have seen a huge leap forward, thanks to the use of a new suit and glove that has greatly improved the fluidity of movements.
But it is mainly thanks to the Unreal Engine 5 program that we have seen an improvement on lip sync which is now much more natural and loose, as well as better management and resolution of lighting and live rendering.
Do you have any new projects in mind to implement with WondAR Studios' technologies?
I am very pleased with the collaboration between Màgina and WondAR Studios. The professionalism and creativity of Paolo & company stimulates me a lot. I would like to open new avenues of application where new realities can give us the wonder, beauty and creativity that we need even more today, to enter a fascinating, attractive and, why not, augmented future.