In this lecture, we learnt how to remove markers from a character’s face in a live footage scene, and how to add texture and corrections that follow the movement of the character.
Degrain/Regrain techniques
Before starting with markers removal from a live footage shot, it is important to degrain our footage so Nuke can read and detect better the pixel information when adding different nodes for cleaning up or tracking techniques. If we use this, then we will need to regrain the plate once we have finish all our changes, so all added elements have the same grain texture and it looks like it has been filmed all in one shot with the same camera and light conditions.
- Simple degrain. We can denoise plates with ‘Merge (minus)‘ followed by a ‘Merge (plus)‘.
- ‘F_ReGRain’ node. This is an alternative to regrain node and it is only available in NukeX. It is more precise than a simple regrain, since it shows less of the patches added for clean up plates.
- ‘DasGrain’ gizmo. This gizmo can be downloaded from Nukepedia where there is also a tutorial on how to use it. We will plug the ‘DasGrain‘ to the original plate and to the denoised plate. Then we plug a ‘Common key‘ gizmo to ‘comp‘ and ‘mark‘ links in ‘DasGrain’. In ‘DasGrain’ node settings, we can set ‘output‘ to desired one (it has different outputs for QC). In the ‘replace‘ tab, we can select the area we want to scan (usually the darkest area), then select ‘activate‘ and then ‘analyse‘. This gizmo is being newly used across VFX companies due to its efficiency and reliability.
Patch changing light techniques
When adding patched to clean up marker in our plate, we need to take care of light changes as the patch could be too obvious:
- First, we can try to correct lighting manually by using a ‘Unpremult‘ node, then ‘Grade‘ by hand in the needed keyframes, and then ‘Premult‘ back. This technique is not recommended as it is time consuming.
- Divide/multiply technique. ‘Blur‘ image (add a lot of blur), then clone the ‘Blur‘ node, and add ‘Merge (divide)‘ to merge both ‘Blur’ nodes. Lastly, ‘Merge (multiply)‘ with background.
- Image Frequency Separation technique. We use ‘Slice Tool‘ gizmo to analyse a specific area of the plate (a face with markers for example), and all frames too (separated gizmo). Then we ‘Blur‘ to see low frequency of image and ‘Merge (from)‘ node to see high frequency. With this, when cloning area with ‘Roto paint‘ to clean markers, we are going to paint only low/high frequencies so the light is not affected (only gamma). This technique is used so light changes do not affect the patched area. With ‘Laplacian‘ node, we could get the same result too. We first need to link with ‘Merge (plus)‘ node to bring back the light and the colours from the original plate, then we ‘Rotopaint’ the part we want, followed by a ‘Blur‘ to add/remove the quantity of light required. Alternatively, we could also ‘Blur‘ and ‘Multiply (divide)‘ to see and correct different values, to then we ‘Merge (multiply)‘ to merge back (like mentioned before).
- Interaction patch technique. Add patch with ‘Roto paint’ with ‘match move‘, then scan original plate with ‘Transform‘, ‘Copy (alpha -> alpha)‘, and ‘Premult‘. Then ‘Merge (multiply)‘ with plate, ‘Regrain‘, and ‘Merge (over)‘ with main plate.
- ‘Curve tool’ node. This is used to add/remove info to the plate (for example, to correct flickering of image). First we start by cropping the info we want by adding ‘Curve tool‘ node, selecting an area, setting ‘curve type‘ as ‘max luma pixel‘ and then click ‘go‘ so it starts to analyse the area. Then, in ‘max or min luma data‘ we click on the icon at the end and then right ‘click + copy + copy links‘. Then we go to ‘grade‘ and ‘paste + paste absolute’ on ‘lift‘ (shadows or min luma data) and ‘gain‘ (luminance or max luma data).
- ‘Roto’ and ‘Transform’ technique. We start with ‘Transform‘ node, followed by a ‘Roto‘ of the part we want, and a ‘Track‘ of the roto. Then we ‘Blur‘ the roto as alpha, ‘Premult‘, and ‘Merge (over)‘ with main plate.
- Clone patch technique. First we denoise the plate so we can ‘Track‘ the markers properly (1 track per marker). Then we copy translate x and centre x to ‘Rotopaint‘ node. We do the patch with clone tool and add ‘Roto‘ over cloned area. Finally, we ‘Filter erode‘, ‘Blur‘, ‘Regrain‘, and ‘Merge (over)‘ to main plate.
- ‘Premult’ and ‘Unpremult’ for paint technique. First, ‘Denoise‘ plate and ‘Track‘ marker. Then copy ‘Roto‘ over marker. ‘Invert‘ roto/mask (like a hole), and ‘Merge (mask)‘ to ‘Shuffle‘. then ‘Blur‘ slightly and link as a mask to ‘Edge blur‘ node which previously was linked to ‘Merge (mask)’ node. Then we ‘Unpremult‘, ‘Copy (alpha -> alpha)‘ from ‘Blur‘ to ‘Premult‘. Lastly, we ‘Regrain‘ (linked to original plate), ‘Premult‘, and ‘Merge (over)‘ to main plate.
- ‘In Paint’ technique. It is nearly the same as the previous technique but, instead of inverting the roto and blur it, this time we use ‘In paint‘ node, which can be tweaked to make the patch blend in.
- ‘UV map’ technique. When using ‘Expression‘ node, R and G channels (X and Y coordinates) have identical values, and just B value is 1, which has no effect on what ST/UV images do. With ‘Expression‘ node, we can ‘Roto paint‘ specific details such as motion blur or warp of an image, and the we connect ‘ST map‘ node to plate. We could also use ‘Grid warp‘ node, but since this is a really heavy tool, it is recommended to avoid this if no needed.
- Vectors technique. As usual, first we ‘Denoise‘ the plate, to then use a ‘Smart vector‘ node. This node could work fine with the default settings, however, it is better to increase ‘detail‘ to achieve a better result and to have less problems with image warp later on. Then we can export this with ‘Write‘ node since smart vectors are really heavy and could slow down the preview. Separately, we remove the markers with ‘Roto paint‘, ‘Filter erode‘, and ‘Blur‘, and we also add a ‘Frame hold‘ node in the reference frame where we are doing the cleaning up. Then we add a ‘Vector distort‘ node that will track the movement of the markers (set ‘output‘ to ‘warped src‘ in this case) following the smart vector map created previously, and then we add a ‘Copy (motion -> motion)‘. Apart, we add a ‘Vector to motion‘ node to add motion blur to the movement of the markers and the we link to to the ‘Copy’ node we added before. Then we add a ‘Vector blur‘ node (with the ‘output‘ as ‘result‘), we ‘Regrain‘, ‘Premult‘ and ‘Merge (over)‘ to main plate. We could also use an ‘ST map’ after the ‘Vector distort’ and in the last one, add ‘output’ as ‘ST map’ instead. This way is better than ‘warped src’, since ‘ST map’ is lighter. Smart vectors can also be used to add texture.
Homework – Face markers clean-up
This week’s homework was to remove the markers of a live footage shot of a girl moving her face. I first tried tracking the markers with a regular ‘Tracker’ node to then link it to the patches made on each marker. This technique is quite straight forward for time consuming since the ‘Tracker’ was also failing to track properly so I had to move the tracker point manually to the correct spot in most of the frames. Also, some of the patches are visible when the girl looks to the sides.
I also tried a different technique, using a ‘Smart vector’ node this time. This technique is really quick if it works fine, however, I am struggling with the distortion of the face when the girl moves her head.
I think I may be doing something wrong as it is distorting the whole image and not just the patches added. I will have to ask Gonzalo in the next class (final result added on Advanced Nuke – Week 9 post)
Final Garage Comp
Since this week I could not go to class in person as I was ill, I did not have the chance to ask for the questions I had regarding the shadows in my garage comp. Therefore, I emailed Gonzalo with a version of my comp attached and my question regarding the shadows being too harsh, and he sent me back a solution to this issue. It looks like I had to add another the ‘Shuffle’ + ‘Blur’ and mask link it to another ‘Grade’ node connected to the main plate, as shown below:
However, I still got the issue of the shadow casting on the wall hole. I tried to add a ‘Merge (stencil)’ node using the previous wall roto I had, however, it was not working as it was cropping the whole wall and not just the hole. I will ask the professor next week about this (final result added on Advanced Nuke – Week 9 post).