Virtual Production Expert Jocelyn Hsu Reveals A ‘Magic Trick’ For Future Film Budgets

Expert advice on where virtual production is headed

Productions are always full of problems. There’s never enough money, enough time, enough people. But for the right kind of person, all those problems become opportunities. To ARwall co-founder and chief product officer Jocelyn Hsu, they can even be fun. 

At ARwall, a technology company that specializes in virtual production products and consulting, Hsu saw an opportunity to help creators think outside the box. For example, filmmakers can pair LED walls with camera tracking to improve upon translight tech that’s been around for a long time. And for Hsu, part of the challenge—and the fun—is working with teams and productions of all sizes.

Recently, we talked to her about maximizing your virtual production budget, what crew you need on set, and where this emerging technology is headed. 

What are the hardest things to do in virtual production? 

What makes us a bit different is that our team is comprised of video game makers and filmmakers. A lot of the other players came from the live event space or other hardware manufacturing, which is excellent because the industry is young and needs players from as many diverse backgrounds as possible. 

But what differentiates ARwall from the rest is we understand that filmmakers are often not developers or game-makers, and building virtual sets requires a lot of different skills. We simplify those tasks so you don't have to code or know how to build a scene that looks photo-real. We can put those things together for you and connect you with partners.

Is creating assets a problem for lower-budget productions?

It depends. What's really cool with being in the Unreal Engine workspace is that they have an extensive library of assets in their marketplace. If you have a lower-budget film and your locations are not that specific or they match the assets that already exist in that marketplace, then it's not too bad. What may get pricey is if you purchase some assets there, but need someone to light it and put it together, which still isn’t that bad because the assets already exist. 

When we worked with Muppets Haunted Mansion, they wanted a location that almost everyone in the world knows: a haunted mansion. They wanted to have little Easter eggs so it wasn't just a replica of an existing ride, but it had to be Muppetized in many ways. And we shot it during COVID, so nobody went to the Haunted Mansion ride to shoot anything.

We reproduced all of that in house. When you're doing assets like that, where everything's bespoke, that can be pricier because now you have a team of artists building those for you.

What do you think the virtual production industry, in general, is doing the hard way?

There's the budget aspect of it and the access.

I think the main difference is when you have a lot of money like if you're at ILM, they're probably innovating on cool things, but from a much larger film perspective. Then the rest of the industry copies that. For example, you’ll find the exact hardware that was used on The Mandalorian is at the top of everyone’s list. 

Filmmaking is like a magic trick. You don't have to spend all that money to achieve the thing.

Our approach is different.  When you're making a film, traditionally you're not going for the most expensive tools possible. Filmmaking is like a magic trick. You don't have to spend all that money to achieve the thing. It just needs to look like you achieved it. We attract a lot of filmmakers who get that because that's what they've done in the past. And then when they're getting into virtual production, they keep that in mind. It should be just another tool because it replaces a good amount of green screen shots, but not all of them.

If you're in a small studio and you have a small wall, are there any kind of tricks you can use? 

We like to look at people's scripts and point them in the best direction. 

If you're working with a small set but you have enough to encapsulate the actor on the LED wall, in order to pull the camera back, you can take the feed from the camera and extend the set in post. You still get the shot with the lighting without extending your LED wall.

Also we still recommend prop pieces on set. But is the actor going to touch it? If they are, then that should be a real set piece. However, for props in front of actors, that's something we can do in an AR sense just like AR filters on your phone. All we have to do is pull the camera feed into the machine that's running the virtual production backdrop, and then append things in front of the actor. A lot of that can also be done in post. 

Can you break down the crew members required for a virtual production set? 

On an ARwall production set, we recommend up to three different people, depending on the size and need of the production.

One of them is the virtual production supervisor. They interact with the rest of the crew to make sure they understand what's needed and communicate that on a higher level.

Then we usually have a virtual production operator or someone who's at the computer running the scenes. They are the game engine expert or the person who makes sure what you're seeing in the virtual scene looks correct—like if the virtual camera movements are right, or if the color matches what the DP and the gaffer team require. 

The third person that we sometimes have is the virtual production technician. They make sure all the hardware, including the LED wall and tracking hardware, are working together correctly. 

Where do you think virtual production is going in the future? 

When virtual production started, it felt like only one-off films would use it, but you’d wonder if a sitcom would. Now it’s becoming more popular for smaller shows. 

One of my favorite examples is Our Flag Means Death on HBO. It's a period piece comedy set on this huge boat that they had to build. It's a pricey thing to do. And when I sat in on a panel, Taika Waititi said they were doing virtual production with an LED wall, because otherwise, they wouldn't be able to afford to make it.

It’s more access for more people, for more creators, to tell the stories they want.

What an amazing thing to hear, that this technology is allowing certain genres and filmmakers to achieve something that previous budgets would not. If they're out on the seas, that location doesn't change too much. They built the boat once and then reuse it again and again. 

In this short span of time, smaller productions can afford to set their characters in locations that normally they couldn’t. It’s more access for more people, for more creators, to tell the stories they want.

What do ADs, DPs, and people who are not on the virtual side of things need to know when they are on a virtual production?

If you're moving equipment, don't scratch up that screen [laughs]. Some virtual production person is going to be very upset with you. 

Ask questions and push the boundaries. A lot of people are not sure what something is, but by the end of the first day, the rest of the crew is in awe when they realize they can do this or that. I think that's where it gets really exciting because we're bringing back the post team onto set again so the collaboration can be there.

When you do green screen, those poor people are working in rooms by themselves. So the collaboration's not quite there and you can see it in actors' performances. Even for amazing award-winning actors, somebody on set didn't tell them what someone in post was going to do and now their eyeline's not matching so it doesn't look right. With this technology, if you see something that could be improved, bring it up because that person is now on set. The feedback is there for the AD, the costumer, the makeup artist, and the actor. It's a huge thing for actors because they can actually see it. 

Productions can move a lot faster with virtual production sets. The gaffer and the DP have more control. We connect the lights to the system to pull the information from the scene. It's almost making your job easier because you don't have to match those colors. The system already knows. It's a lot of different innovations that are waiting to be implemented.

With Showrunner, the tech is this great little centerpiece that can help people collaborate better because they'll be able to call it up on set or do something straight on the tablet to affect that change.

This interview has been lightly edited and condensed for clarity.