with XSeg model you can train your own mask segmentator of dst(and src) faces
that will be used in merger for whole_face.
Instead of using a pretrained model (which does not exist),
you control which part of faces should be masked.
Workflow is not easy, but at the moment it is the best solution
for obtaining the best quality of whole_face's deepfakes using minimum effort
without rotoscoping in AfterEffects.
new scripts:
XSeg) data_dst edit.bat
XSeg) data_dst merge.bat
XSeg) data_dst split.bat
XSeg) data_src edit.bat
XSeg) data_src merge.bat
XSeg) data_src split.bat
XSeg) train.bat
Usage:
unpack dst faceset if packed
run XSeg) data_dst split.bat
this scripts extracts (previously saved) .json data from jpg faces to use in label tool.
run XSeg) data_dst edit.bat
new tool 'labelme' is used
use polygon (CTRL-N) to mask the face
name polygon "1" (one symbol) as include polygon
name polygon "0" (one symbol) as exclude polygon
'exclude polygons' will be applied after all 'include polygons'
Hot keys:
ctrl-N create polygon
ctrl-J edit polygon
A/D navigate between frames
ctrl + mousewheel image zoom
mousewheel vertical scroll
alt+mousewheel horizontal scroll
repeat for 10/50/100 faces,
you don't need to mask every frame of dst,
only frames where the face is different significantly,
for example:
closed eyes
changed head direction
changed light
the more various faces you mask, the more quality you will get
Start masking from the upper left area and follow the clockwise direction.
Keep the same logic of masking for all frames, for example:
the same approximated jaw line of the side faces, where the jaw is not visible
the same hair line
Mask the obstructions using polygon with name "0".
run XSeg) data_dst merge.bat
this script merges .json data of polygons into jpg faces,
therefore faceset can be sorted or packed as usual.
run XSeg) train.bat
train the model
Check the faces of 'XSeg dst faces' preview.
if some faces have wrong or glitchy mask, then repeat steps:
split
run edit
find these glitchy faces and mask them
merge
train further or restart training from scratch
Restart training of XSeg model is only possible by deleting all 'model\XSeg_*' files.
If you want to get the mask of the predicted face in merger,
you should repeat the same steps for src faceset.
New mask modes available in merger for whole_face:
XSeg-prd - XSeg mask of predicted face -> faces from src faceset should be labeled
XSeg-dst - XSeg mask of dst face -> faces from dst faceset should be labeled
XSeg-prd*XSeg-dst - the smallest area of both
if workspace\model folder contains trained XSeg model, then merger will use it,
otherwise you will get transparent mask by using XSeg-* modes.
Some screenshots:
label tool: https://i.imgur.com/aY6QGw1.jpg
trainer : https://i.imgur.com/NM1Kn3s.jpg
merger : https://i.imgur.com/glUzFQ8.jpg
example of the fake using 13 segmented dst faces
: https://i.imgur.com/wmvyizU.gifv
for every batch_size*16 samples,
model collects the samples with the highest error and learns them again
therefore hard samples will be trained more often
Basic usage instruction: https://i.imgur.com/w7LkId2.jpg
'whole_face' requires skill in Adobe After Effects.
For using whole_face you have to extract whole_face's by using
4) data_src extract whole_face
and
5) data_dst extract whole_face
Images will be extracted in 512 resolution, so they can be used for regular full_face's and half_face's.
'whole_face' covers whole area of face include forehead in training square,
but training mask is still 'full_face'
therefore it requires manual final masking and composing in Adobe After Effects.
added option 'masked_training'
This option is available only for 'whole_face' type.
Default is ON.
Masked training clips training area to full_face mask,
thus network will train the faces properly.
When the face is trained enough, disable this option to train all area of the frame.
Merge with 'raw-rgb' mode, then use Adobe After Effects to manually mask, tune color, and compose whole face include forehead.
added option Eyes priority (y/n)
fix eye problems during training ( especially on HD architectures )
by forcing the neural network to train eyes with higher priority
before/after https://i.imgur.com/YQHOuSR.jpg
It does not guarantee the right eye direction.
SAEHD:
added new option
GAN power 0.0 .. 10.0
Train the network in Generative Adversarial manner.
Forces the neural network to learn small details of the face.
You can enable/disable this option at any time,
but better to enable it when the network is trained enough.
Typical value is 1.0
GAN power with pretrain mode will not work.
Example of enabling GAN on 81k iters +5k iters
https://i.imgur.com/OdXHLhU.jpghttps://i.imgur.com/CYAJmJx.jpg
dfhd: default Decoder dimensions are now 48
the preview for 256 res is now correctly displayed
fixed model naming/renaming/removing
Improvements for those involved in post-processing in AfterEffects:
Codec is reverted back to x264 in order to properly use in AfterEffects and video players.
Merger now always outputs the mask to workspace\data_dst\merged_mask
removed raw modes except raw-rgb
raw-rgb mode now outputs selected face mask_mode (before square mask)
'export alpha mask' button is replaced by 'show alpha mask'.
You can view the alpha mask without recompute the frames.
8) 'merged *.bat' now also output 'result_mask.' video file.
8) 'merged lossless' now uses x264 lossless codec (before PNG codec)
result_mask video file is always lossless.
Thus you can use result_mask video file as mask layer in the AfterEffects.
Removed the wait at first launch for most graphics cards.
Increased speed of training by 10-20%, but you have to retrain all models from scratch.
SAEHD:
added option 'use float16'
Experimental option. Reduces the model size by half.
Increases the speed of training.
Decreases the accuracy of the model.
The model may collapse or not train.
Model may not learn the mask in large resolutions.
true_face_training option is replaced by
"True face power". 0.0000 .. 1.0
Experimental option. Discriminates the result face to be more like the src face. Higher value - stronger discrimination.
Comparison - https://i.imgur.com/czScS9q.png
