Trainer: added option for all models

Enable autobackup? (y/n ?:help skip:%s) :
Autobackup model files with preview every hour for last 15 hours. Latest backup located in model/<>_autobackups/01

SAE: added option only for CUDA builds:
Enable gradient clipping? (y/n, ?:help skip:%s) :
Gradient clipping reduces chance of model collapse, sacrificing speed of training.
This commit is contained in:
iperov 2019-06-20 10:42:55 +04:00
commit 8484060e01
14 changed files with 210 additions and 80 deletions

Binary file not shown.