Nira's VFX Blog
A journey through my master

Tag: Blend Shape

Categories
Maya VFX Fundamentals

Week 9: Speech Lip Sync in Maya

In this class, we learnt to synchronise the mouth and facial expressions of our face model with a short speech that we selected.

First, to capture a video from internet (YouTube) in this case to use it as a reference to build up our speech lip sync, we used ‘OBS Studio’. With this programme we can record the screen and sound of our computer’s desktop and then edit the length and export in the desired format in Adobe After Effects, or Adobe Premiere Pro. The sequence that I chose shows Charles Xavier talking to Magneto in X-Men Apocalypse (Movie Scenes, 2021).

Magneto’s Final Talk With Charles Xavier | X-Men Apocalypse (2016), (Movie Scenes, 2021)

Then, we imported it to Maya as an ‘Image Plane’. It is important to have set the timeline with the same fps (frames per second) to our clip, otherwise, it will not be in sync with the audio. In my case, I exported my clip at 30 fps so set Maya with the same value. To import the audio, we right click in the timeline and import it from there. If we want to preview the clip with the audio to double check that they are in sync, we can use ‘Playblast’ feature which will show a low-resolution preview. 

Image plane and audio imported to Maya

Once we had our reference clip set, we started to animate the mouth creating new targets in the blend shape previously created to fit the mouth shape to each sound of the speech. Then, using keyframing we set the exact movements we wanted and smooth them editing and adjusting the key frames in the ‘Graph Editor’. A useful reference to build the mouth shape depending on the sound of each syllable, I took as a reference an online image from Preston Blair Phoneme Series (Martin, 2018).

Preston Blair phoneme serie (Martin, 2018)

I did not create all the phonemes as not all of them were used and I actually blended 2 or 3 phonemes at the same time to create a different phoneme with this. I also animated the tongue, the eye brows, the jaw, the head rotate, and the neck (Addam’s apple). Below there is a sequence of screen captures of all the movements together and the targets that have been used.

‘I’ phoneme
‘Fee’ phoneme
‘A’ phoneme
‘O’ phoneme
‘P’ phoneme
‘Grea’ phoneme
‘Fo’ phoneme
‘Oo’ phoneme
‘M’ phoneme
‘B’ phoneme
‘L’ phoneme – can see the tongue in this one
‘F’ phoneme

I really enjoyed modelling and animating the facial expressions. I definitely need to improve and refine the animation part as some of the phonemes are not as polished as I would have liked but to be my first time animating a face and lip synching a speech I think it looks really good. I feel like this is an area that I would like to explore further.

References

Martin, G. C., 2018. Preston Blair phoneme series. [Online] Available at: http://www.garycmartin.com/mouth_shapes.html [Accessed 12 December 2022].

Movie Scenes, 2021. Magneto’s Final Talk With Charles Xavier | X-Men Apocalypse (2016). [Online] Available at: https://www.youtube.com/watch?v=1gZqgfiWDh4 [Accessed 28 November 2022].

Categories
Maya VFX Fundamentals

Week 8: Facial Expressions with Blend Shapes in Maya

In this week’s lecture, we learnt how to edit blend shapes to create facial expressions and how to make those actions interact with each other so we achieve a natural movement when animating.

Creating blend shapes and targets inside these, we can animate our face’s expressions. One by one I created the expressions as full smile, half smile, frown, half frown, blinking eyes, and open mouth. I built up the expressions using the brush and relax tools. I also added a correction in the mouth for when it is opened to relax the sides of it and make it look more realistic. Since this correction is only needed when the mouth opens, I set the mouth correction target as a driven key and the jaw joint as driver and key framed the animation.

Mouth correction for open mouth movement using ‘set driven key’ tool
Mouth correction for closed mouth movement using ‘set driven key’ tool
Closed eyes target
Frown target
Right eye brow raised target
Left eye brow raised target
Smile target
Right half smile target
Left eye brow raised mixed with left half smile target
Right eye brow raised mixed with right half smile target
Open mouth animation with mouth correction target
Smile target with open mouth animation
Frown and smile targets with open mouth animation
Final expressions
Final expressions
Mouth opening with final frown animation experimentation

Initially, I struggle a bit creating the facial expressions as sometimes I forgot to switch the ‘Edit’ button of the target I was manipulating at the time and I had some errors when animating that affected previous targets. Also, I seemed to have a duplicated head node in the background and some other duplicated ‘set’ nodes that were giving me an error message all the time. After asking Nick for help, he found this duplicated nodes in the ‘Node Editor’, and after delete them, my face model started to work better. I had to delete all the targets and the blend shape I had and start from scratch. I did this numerous times before finding the real error, and it was really time consuming, but at the end of the day, it was a really good practice and I learnt how to solve the error myself for future projects.

Categories
Maya VFX Fundamentals

Week 7: Facial Animation Set Up, Hierarchies and Rigging

In this session, we reminded how to create blend shapes to animate the facial expressions and how to create a ‘rig’ or ‘skeleton’ to animated the head and the mouth of our model.

Using the ‘Shape Editor’ tool, we can create a blend shape or shape variation in order to set the facial expressions of our model. On each blend shape, we need to add ‘target’ points with which we could create our movements or reshapes, for example the eyes opening and closing, the mouth smile, or the eye brows frown.

In order to animate the head and the mouth opening, we created a ‘rig’ or ‘skeleton’ that will determine the joints of the neck and jaw. After setting the rig, we bound the skin of the model to our rig and painted the skin weights to add the influence parts of our model (the parts the will be more influence by the rig movement).

Lastly, we created the model’s set of teeth and added them to the rig influence.

Closed eyes
Open eyes
Mixed blend shapes – Smile with open eyes
Mixed blend shapes – Smile with closed eyes
Mixed blend shapes – Half smile with open eyes
Rigging
Jaw painted weights
Head painted wights
Neck painted weights
Mouth opening with adjustment of painted weights
Mouth opening with adjustment of painted weights
Mouth opening with adjustment of painted weights
Teeth wireframe
Teeth model
Facial expressions sequence
Mouth opening sequence

I struggled a bit with the painting of the weights to open the mouth of the model. I had to adjust my mesh with the ‘soft brush’ and ‘relax’ tools so it started to respond appropriately.