This week our team leader created a head model for digital human, and my goal is to import this model into the "Test Ground" that I set up last week for connecting our model with Convai. The problem of lip sync offered by Convai works on the head model, but the head tracking and eye blink part did not work, so I need to create own script to make it work on the head model.
Current Head Model
Task 1: Create a new head tracking script Since what I have now is the head part only so moving the whole object not only head works fine
This script is for rotating the whole gameobject with camera movement, so currently it's fine, if we have more that head model or require better performance later, it needs to upgrade.
Task 2: Eye blink while talking
Since our goal is to make a realistic digital human avator so it should blink eyes while talking(just like human). This is a challenge for me since I never used blendshape before. The first thing for me to do is to search how blendshape works, and how blendshape could make the head blink. Second, I need to find the proper blendshape index for left and right eye. After several test and fix, here is the final version of my blinkingeye script, and I write comment in each line as reminder for me to look in future development.
Here is the script part public class BlinkDuringTalk : MonoBehaviour {
public Animator animator; // Animator controller
public SkinnedMeshRenderer skinnedMeshRenderer; // The SkinnedMeshRenderer component of the model public int blinkLeftIndex = 11; // Index of the BlendShape left eye public int blinkRightIndex = 12; // Index of the BlendShape right eye
void Update() { // Check if the "Talk" animation is playing if (animator.GetCurrentAnimatorStateInfo(0).IsName("Talk")) { // Increase the blink frequency to test if the coroutine is triggered if (Random.Range(0, 100) < 50) // 50% chance to trigger blinking each frame { StartCoroutine(Blink()); } } } IEnumerator Blink() { Debug.Log("Blink started"); // Log message started // Start blinking skinnedMeshRenderer.SetBlendShapeWeight(blinkLeftIndex, 100); skinnedMeshRenderer.SetBlendShapeWeight(blinkRightIndex, 100); yield return new WaitForSeconds(0.1f); // Duration of the blink Debug.Log("Blink ended"); // Log message ended // End blinking skinnedMeshRenderer.SetBlendShapeWeight(blinkLeftIndex, 0); skinnedMeshRenderer.SetBlendShapeWeight(blinkRightIndex, 0); } }
This is the first try for me on blendshapes, it has a lot of part could be update and i will keep my mind on the needs to fix script.
The next step and self-reflection
Currently the head tracking works but not what we want, so I need to try to apply animation rigging for control the eye ball of head model and let the rotation of eye to drive the head.
And we want the head to have different emotion state and based on its current emotion state to change the facial representation. For this propose, we come up the idea of using scriptable object to store the value for head. And also is a new thing for me which I need to look up for the methods.
During the step up, I realized that I designed without much research foudation, and I need to look more field like psychology side to understand the meaning of value I need to set up for the script.