3ds Max Plugin Integration with NVIDIA ACE & Audio2Face

This guide provides a complete step-by-step installation and integration setup for connecting a 3ds Max plugin with NVIDIA ACE and Audio2Face (A2F) using Docker and gRPC.

High-Level Workflow

• audio2face 3dsMax2011 Plugin → Sends audio, face mesh, and emotion parameters to A2F Controller via gRPC (port 52000).
• A2F Controller → Passes data to Audio2Face (port 50000).
• Audio2Face → Processes the audio, generates blendshapes, and sends results back to the A2F Controller (port 51000).
• A2F Controller → Returns processed animation data to the 3ds Max plugin.
• audio2face 3dsMax2011 Plugin → Stores animation as keyframes, syncing with the 3ds Max timeline and playing audio with the animation.
• FaceFX 3dsMax2011 Plugin → Exports the animation in .fxa format.

Setting Up the Environment

Step 1: Install & Configure Docker on Windows

1. Download & Install Docker Desktop.
2. Enable "Use the WSL 2 based engine" during installation and restart Docker.

Step 2: Install WSL2 & Ubuntu

wsl --install
wsl --set-default-version 2

Then, open Ubuntu and update packages:

sudo apt update && sudo apt upgrade -y

Step 3: Install NVIDIA Drivers & Container Toolkit

Check NVIDIA Driver Version:

nvidia-smi

If missing or outdated, install:

sudo apt install nvidia-driver-535

Install NVIDIA Container Toolkit:

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update
sudo apt install -y nvidia-container-toolkit

Developing the 3ds Max Plugin

Step 1: MaxScript - Call Python

The following MaxScript launches Python to send audio and face mesh data to A2F:

global A2F_Client = "C:\\path\\to\\a2f_client.py"
fn runA2F audioFile configYml ipPort = (
    local cmd = "python \"" + A2F_Client + "\" \"" + audioFile + "\" \"" + configYml + "\" -u " + ipPort
    shellLaunch "cmd.exe" ("/c " + cmd)
    format "A2F Process Started... \n"
)
runA2F "C:\\path\\to\\audio.wav" "C:\\path\\to\\config.yml" "127.0.0.1:52000"

Step 2: Python - gRPC Client

Install dependencies:

pip install grpcio grpcio-tools numpy pandas

Python gRPC Client:

import grpc
import csv
import sys
import os
import a2f_pb2
import a2f_pb2_grpc

def send_audio(audio_file, config_file, ip_port):
    # Validate input files
    if not os.path.exists(audio_file):
        print(f"Error: Audio file '{audio_file}' not found.")
        return
    if not os.path.exists(config_file):
        print(f"Error: Config file '{config_file}' not found.")
        return
    
    # Establish gRPC channel with timeout and error handling
    try:
        channel = grpc.insecure_channel(ip_port)
        stub = a2f_pb2_grpc.A2FServiceStub(channel)
    except grpc.RpcError as e:
        print(f"gRPC Error: {e}")
        return
    
    # Read audio file
    try:
        with open(audio_file, "rb") as f:
            audio_data = f.read()
    except Exception as e:
        print(f"Error reading audio file: {e}")
        return
    
    # Create request and send to A2F Controller
    request = a2f_pb2.A2FRequest(audio=audio_data, config=config_file)
    
    try:
        response = stub.ProcessAudio(request, timeout=10)  # 10s timeout
    except grpc.RpcError as e:
        print(f"gRPC Processing Error: {e}")
        return
    
    # Ensure output directory exists
    output_csv = "C:\\path\\to\\output_keyframes.csv"
    os.makedirs(os.path.dirname(output_csv), exist_ok=True)
    
    # Save keyframes to CSV
    try:
        with open(output_csv, "w", newline="") as csvfile:
            writer = csv.writer(csvfile)
            writer.writerow(["frame", "blendshape", "value"])
            for frame in response.keyframes:
                writer.writerow([frame.time, frame.name, frame.value])
        print(f"Keyframes saved to {output_csv}")
    except Exception as e:
        print(f"Error writing CSV: {e}")
    
if __name__ == "__main__":
    if len(sys.argv) != 4:
        print("Usage: python client.py <audio_file> <config_file> <ip:port>")
    else:
        send_audio(sys.argv[1], sys.argv[2], sys.argv[3])

Step 3: MaxScript - Apply Keyframes

Apply received keyframes to the 3D model:

fn applyKeyframes csvFile = (
    local f = openFile csvFile
    while not eof f do (
        local line = readDelimitedString f ","
        local frameNum = line[1] as integer
        local blendshapeName = line[2] as string
        local value = line[3] as float

        local target = $FaceMesh.modifiers[#Morpher]
        if target != undefined do (
            local channelIndex = findItem target.channelNames blendshapeName
            if channelIndex > 0 do (
                at time frameNum target[blendshapeName].value = value
            )
        )
    )
    close f
    format "Keyframes Applied! \n"
)
applyKeyframes "C:\\path\\to\\output_keyframes.csv"

Summary

• MaxScript launches Python to send audio & face mesh data to A2F.
• Python gRPC Client sends the data and receives blendshape keyframes.
• MaxScript applies the received keyframes to the 3D model in 3ds Max.

This setup automates facial animation in 3ds Max using NVIDIA ACE & Audio2Face! 🎭