wav2lip-384x384 训练

崎山小鹿 · 发表于 2024-11-14 12:54:18

Wav2Lip是一种基于深度学习的人物口型生成技术，它可以将音频信号转化为对应的人物口型动画。简单来说，就是通过分析音频信号中的语音信息，从而生成出与语音内容相匹配的口型动画。这一技术的出现，极大地提高了虚拟数字人的逼真程度，使得观众能够获得更加真实的视听体验
选择原视频和原音频
微信截图_20241115092747.png

点击“开始生成”

看效果

还是非常优秀的。

生成视频耗时比较长，面部有点模糊，要求视频人物不说话会支持得比较好。

判断音频和唇形在某个共同参数空间下的相似性。

推理过程

1）对输入语音提取Mel特征，得到语音特征块
2）对全脸+去掉下半张脸（6* 96* 96）两组人脸提取图片特征
3）将上面两种特征输入到wav2lip网络，生成带口型的人脸
4）将带口型的人脸贴回原图，逐帧写成纯图像视频
5）用ffmpeg合成带语音的视频

Wav2Lip是一种深度学习模型，其核心思想是将音频波形直接转换为面部动画。该模型基于生成对抗网络（GAN）设计，包含生成器和判别器两个主要部分。生成器的任务是根据输入的音频波形生成逼真的面部动画，而判别器的目标是区分生成的动画与真实的面部动画。

生成器G GG负责生成目标口型的人脸图像，由三部分组成：身份编码器（Identity Encoder）、语音编码器（Speech Encoder）和人脸解码器（Face Decoder），这三部分均是由堆叠的2D卷积层组成。概括来说，生成器是一个2D卷积的编码器-解码器结构。

身份编码器的把随机参考帧R RR与先验姿势P PP(下半部分被mask的目标脸)按通道维度拼接作为输入，编码身份特征。先验姿势帧的下半部分被mask，但是提供了目标人脸的姿态信息；参考帧则包含目标人脸的完整外观，用于唇部形状和运动的合成。
语音编码器用于编码输入的语音片段
人脸解码器以编码后的音频特征与身份特征的拼接为输入，通过反卷积进行上采样，重建人脸图像。
生成器通过最小化生成帧L g L_{g}L
g

与真实帧L G L_{G}L
G

之间的L1重构损失来进行训练。

Easy-Wav2Lip是Wav2Lip的改进版本，在设计上更为简洁，执行速度更快，同时生成的视频效果更加逼真。Wav2Lip技术可以让视频中的人物根据输入的音频生成匹配的唇形动作，从而实现口型与语音同步的效果。这项技术不仅适用于静态图像，还能够对动态视频进行处理，生成与目标语音同步的视频输出。

项目源码下载地址：https://github.com/nghiakvnvsd/wav2lip384

参考：
https://blog.csdn.net/jiafeier_555/article/details/131404553
https://blog.csdn.net/weixin_42111770/article/details/134691558
warlip384训练视频

崎山小鹿 · 发表于 2024-11-15 13:04:43

项目地址：https://github.com/zhangbo2008/wav2lip384_my2/tree/main
运行：
python 0.py

ModuleNotFoundError: No module named 'cv2' （安装cv2）
pip install opencv-python

ModuleNotFoundError: No module named 'pytorch_lightning'，
pip install pytorch_lightning==2.2
版本查询参考：https://blog.csdn.net/weixin_43135178/article/details/134719948

OSError: [WinError 127] 找不到指定的程序。 Error loading "C:\Users\Administrator\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\torch\lib\torch_python.dll" or one of its dependencies.
是因为版本冲突，重新安装torch
pip install torch==2.0.1

Traceback (most recent call last):
File "G:\wav2lip384_my2-main\0.py", line 11, in <module>
raise FileNotFoundError('Save the s3fd model to face_detection/detection/sfd/s3fd.pth \
FileNotFoundError: Save the s3fd model to face_detection/detection/sfd/s3fd.pth before running this script!缺文件就去https://github.com/Rudrabha/Wav2Lip下载.
缺失人脸预测模型，下载地址如下：
https://www.adrianbulat.com/down ... s3fd-619a316812.pth

raise AssertionError("Torch not compiled with CUDA enabled")，问题出现在Pytorch和CUDA版本不兼容的问题上。
在python下输入如下代码查询torch的版本以及cuda的使用情况。
import torch
print(torch.__version__)
print(torch.cuda.is_available())

>>> print(torch.cuda.is_available())
False
说明cuda没有使用上。
先查看cuda安装有没有问题：nvcc -V

安装cuda
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

安装成功
Installing collected packages: torchvision, torchaudio
Successfully installed torchaudio-2.0.2+cu118 torchvision-0.15.2+cu118
但是coda依然无法使用

切换python版本
pyenv local 3.8.0

将你的包requirements.txt内容替换如下即可
原内容：
librosa==0.7.0
numpy==1.17.1
opencv-contrib-python>=4.2.0.34
opencv-python==4.1.0.25
torch==1.1.0
torchvision==0.3.0
tqdm==4.45.0
numba==0.48

替换为：
librosa==0.9.1
numpy>=1.17.3
opencv-contrib-python>=4.4.0.44
opencv-python>=4.4.0.44
torch>=1.7.1
torchvision>=0.8.2
tqdm>=4.45.0
numba>=0.48

安装
.\py311\python.exe -m pip install -r requirements.txt

安装成功：
Installing collected packages: mpmath, zipp, urllib3, typing-extensions, threadpoolctl, sympy, pycparser, platformdirs, pillow, packaging, numpy, networkx, MarkupSafe, llvmlite, joblib, idna, fsspec, filelock, decorator, colorama, charset-normalizer, certifi, audioread, tqdm, scipy, requests, opencv-python, opencv-contrib-python, jinja2, importlib-resources, importlib-metadata, cffi, torch, soundfile, scikit-learn, pooch, numba, torchvision, resampy, librosa
Successfully installed MarkupSafe-2.1.5 audioread-3.0.1 certifi-2024.8.30 cffi-1.17.1 charset-normalizer-3.4.0 colorama-0.4.6 decorator-5.1.1 filelock-3.16.1 fsspec-2024.10.0 idna-3.10 importlib-metadata-8.5.0 importlib-resources-6.4.5 jinja2-3.1.4 joblib-1.4.2 librosa-0.9.1 llvmlite-0.41.1 mpmath-1.3.0 networkx-3.1 numba-0.58.1 numpy-1.24.4 opencv-contrib-python-4.10.0.84 opencv-python-4.10.0.84 packaging-24.2 pillow-10.4.0 platformdirs-4.3.6 pooch-1.8.2 pycparser-2.22 requests-2.32.3 resampy-0.4.3 scikit-learn-1.3.2 scipy-1.10.1 soundfile-0.12.1 sympy-1.13.3 threadpoolctl-3.5.0 torch-2.4.1 torchvision-0.19.1 tqdm-4.67.0 typing-extensions-4.12.2 urllib3-2.2.3 zipp-3.20.2

安装pytorch_lightning
.\py311\python.exe -m pip install pytorch_lightning==2.2

ModuleNotFoundError: No module named 'madgrad'

.\py311\python.exe -m pip install madgrad

崎山小鹿 · 发表于 2024-11-16 14:08:00

数据准备
dirname = vfile.split('/')[-2] 获取视频文件所在的父级目录名称
在windows下要改为：dirname = vfile.split('\\')[-2]

崎山小鹿 · 发表于 2024-11-18 10:05:18

崎山小鹿 · 发表于 2024-11-18 10:22:47

2.如何继续训练
继续训练的关键代码：
args.syncnet_checkpoint_path='weight\syncnet\ex\syncnet_checkpoint_384_20_000021500_2024-11-18.pth'
args.checkpoint_path='weight\wav\ex_wav2lip_margin\gen_last_wav128_1e4.pth'  #继续训练
args.data_root='preprocessed_root/data_train'

Load checkpoint from: weight\syncnet\ex\syncnet_checkpoint_384_20_000021500_2024-11-18.pth
G:\wav2lip384_my2-main\2.py:684: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytor ... md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(checkpoint_path)
total wav2lip trainable params 159087803
total disc trainable params 43082817
total syncnet  params 65054464
Starting Epoch: 0
G:\wav2lip384_my2-main\2.py:406: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
G:\wav2lip384_my2-main\2.py:429: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
Step 1 | L1: 2.40438 | Sync_wt: 0.0030 Sync: 0.421151, Percep: 0.703886 | Fake: 0.682866, Real: 0.709179 | Load: 0.208103, Train: 3.14551
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
...
Starting Epoch: 44
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 221 | L1: 0.408767 | Sync_wt: 0.0300 Sync: 0.973213, Percep: 0.224766 | Fake: 1.93283, Real: 0.0268605 | Load: 0.17092, Train: 2.21316
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 222 | L1: 0.477655 | Sync_wt: 0.0300 Sync: 1.43658, Percep: 2.60373 | Fake: 0.971099, Real: 0.876769 | Load: 0.187253, Train: 1.71426
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 223 | L1: 0.491983 | Sync_wt: 0.0300 Sync: 1.46766, Percep: 3.08503 | Fake: 0.654007, Real: 0.707941 | Load: 0.183522, Train: 1.88717
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 224 | L1: 0.503687 | Sync_wt: 0.0300 Sync: 1.37013, Percep: 3.26725 | Fake: 0.496582, Real: 0.618142 | Load: 0.170331, Train: 1.87882
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 225 | L1: 0.506882 | Sync_wt: 0.0300 Sync: 1.16493, Percep: 3.3811 | Fake: 0.402043, Real: 0.513971 | Load: 0.169339, Train: 1.88462
torch.Size([1, 3, 5, 384, 384])
torch.Size([1, 3, 5, 192, 384])

...
Starting Epoch: 473
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2366 | L1: 0.231807 | Sync_wt: 0.0300 Sync: 1.22511, Percep: 1.69847 | Fake: 0.206287, Real: 0.589591 | Load: 0.176917, Train: 2.5686
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2367 | L1: 0.205227 | Sync_wt: 0.0300 Sync: 1.29311, Percep: 2.06025 | Fake: 0.435907, Real: 0.298213 | Load: 0.199886, Train: 1.97014
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2368 | L1: 0.220921 | Sync_wt: 0.0300 Sync: 0.943628, Percep: 2.31971 | Fake: 0.321806, Real: 0.293943 | Load: 0.19954, Train: 2.16911
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2369 | L1: 0.22782 | Sync_wt: 0.0300 Sync: 1.16881, Percep: 2.55586 | Fake: 0.251937, Real: 0.311174 | Load: 0.177899, Train: 2.17077
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2370 | L1: 0.250215 | Sync_wt: 0.0300 Sync: 1.4198, Percep: 2.47995 | Fake: 0.226859, Real: 0.291268 | Load: 0.173385, Train: 2.11635
torch.Size([1, 3, 5, 384, 384])
torch.Size([1, 3, 5, 192, 384])

Starting Epoch: 569
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2846 | L1: 0.225897 | Sync_wt: 0.0300 Sync: 0.53349, Percep: 2.02565 | Fake: 0.151527, Real: 0.0176994 | Load: 0.176329, Train: 2.2614
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2847 | L1: 0.23254 | Sync_wt: 0.0300 Sync: 0.889151, Percep: 2.03761 | Fake: 0.186591, Real: 0.0162578 | Load: 0.190352, Train: 1.75833
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2848 | L1: 0.251053 | Sync_wt: 0.0300 Sync: 0.725673, Percep: 2.00203 | Fake: 0.228083, Real: 0.015268 | Load: 0.177649, Train: 1.92353
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2849 | L1: 0.251991 | Sync_wt: 0.0300 Sync: 0.698621, Percep: 2.0589 | Fake: 0.241014, Real: 0.0180526 | Load: 0.179897, Train: 1.88287
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 2850 | L1: 0.253426 | Sync_wt: 0.0300 Sync: 0.615678, Percep: 2.68241 | Fake: 0.194633, Real: 0.132936 | Load: 0.185898, Train: 1.98481
torch.Size([1, 3, 5, 384, 384])
torch.Size([1, 3, 5, 192, 384])

Starting Epoch: 758
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 3791 | L1: 0.206628 | Sync_wt: 0.0300 Sync: 0.935533, Percep: 5.11073 | Fake: 0.00862205, Real: 0.00432247 | Load: 0.16981, Train: 2.25067
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 3792 | L1: 0.250087 | Sync_wt: 0.0300 Sync: 0.695988, Percep: 4.94466 | Fake: 0.00982878, Real: 0.00432828 | Load: 0.212077, Train: 1.79654
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 3793 | L1: 0.232677 | Sync_wt: 0.0300 Sync: 0.613124, Percep: 4.96901 | Fake: 0.00963348, Real: 0.00433 | Load: 0.1809, Train: 1.96615
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 3794 | L1: 0.23422 | Sync_wt: 0.0300 Sync: 0.582902, Percep: 5.00178 | Fake: 0.00941184, Real: 0.00435563 | Load: 0.190866, Train: 1.94068
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 3795 | L1: 0.240213 | Sync_wt: 0.0300 Sync: 0.695015, Percep: 5.009 | Fake: 0.00937021, Real: 0.00435027 | Load: 0.182902, Train: 1.92828
torch.Size([1, 3, 5, 384, 384])
torch.Size([1, 3, 5, 192, 384])

Starting Epoch: 929
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 4646 | L1: 0.235182 | Sync_wt: 0.0300 Sync: 1.34508, Percep: 1.38635 | Fake: 0.795776, Real: 0.277558 | Load: 0.165917, Train: 2.22869
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 4647 | L1: 0.228566 | Sync_wt: 0.0300 Sync: 0.868533, Percep: 1.33973 | Fake: 0.59392, Real: 0.170588 | Load: 0.186512, Train: 1.7112
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 4648 | L1: 0.226583 | Sync_wt: 0.0300 Sync: 1.14048, Percep: 1.73176 | Fake: 0.458232, Real: 0.278667 | Load: 0.176913, Train: 1.89952
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])
Step 4649 | L1: 0.229176 | Sync_wt: 0.0300 Sync: 0.885194, Percep: 2.20822 | Fake: 0.353763, Real: 0.230063 | Load: 0.162894, Train: 1.88558
torch.Size([2, 3, 5, 384, 384])
torch.Size([2, 3, 5, 192, 384])

崎山小鹿 · 发表于 2024-11-19 18:22:48

3.推理
遇到两个问题
1.百叶窗问题
2.嘴巴不动的问题
3.脸对齐问题

崎山小鹿 · 发表于 2024-11-19 20:03:28

项目地址：https://github.com/primepake/wav2lip_288x288
错误：
from torchsummary import summary
ModuleNotFoundError: No module named 'torchsummary'
torchsummary 可以做Pytorch可视化，输出网络相关信息
解决：
.\py311\python.exe -m pip install torchsummary

问题：
Epoch time: 9.189886331558228
Starting Epoch: 35
use_cuda: True
use_cuda: True
use_cuda: True
use_cuda: True

崎山小鹿 · 发表于 2024-11-20 09:02:41

Wav2Lip 原项目：https://github.com/Rudrabha/Wav2Lip

Wav2Lip 在Sync Labs免费托管 https://platform.sync.so/

colab服务器代码：
https://colab.research.google.co ... rollTo=KoVGMtjRZfeR

https://colab.research.google.co ... rollTo=jR5utmDMcSZY

gan模型下载：https://iiitaphyd-my.sharepoint. ... p%5FModels&ga=1

报错：
Wav2Lip 报错TypeError: mel() takes 0 positional arguments but 2 positional arguments (and 3 keyword-only arguments) were given
解决：
应该是librosa版本的问题，但是老版本的安装上不上，我一直很苦恼怎么解决这个问题
修复此问题不走：

修改依赖版本号：
librosa==0.10.1
numpy==1.24.3
opencv-contrib-python>=4.2.0.34
opencv-python>=4.7.0.72
torch==1.11.0
torchvision==0.12.0
tqdm==4.45.0
numba==0.59.0
修改audio.py 第一百行代码：
return librosa.filters.mel(sr=hp.sample_rate, n_fft=hp.n_fft, n_mels=hp.num_mels,
fmin=hp.fmin, fmax=hp.fmax)
调整参数在运行就没问题了。

前面预处理数据集
在video中准备好要训练得源视频文件
python preprocess.py --data_root video/ --preprocessed_root filelists/
将 LRS2 文件列表（train、val、test）.txt文件放在文件夹中filelists/

第一步训练专家鉴别器
python color_syncnet_train.py --data_root lrs2_preprocessed/ --checkpoint_dir <folder_to_save_checkpoints>

第二步训练 Wav2Lip 模型
python wav2lip_train.py --data_root lrs2_preprocessed/ --checkpoint_dir <folder_to_save_checkpoints> --syncnet_checkpoint_path <path_to_expert_disc_checkpoint>
要使用视觉质量鉴别器进行训练，您应该hq_wav2lip_train.py

第三部推理
您可以将任意视频与任意音频进行口型同步
python inference.py --checkpoint_path <ckpt> --face <video.mp4> --audio <an-audio-source>

结果（默认）保存在中results/result_voice.mp4。您可以将其指定为参数，类似于其他几个可用选项。音频源可以是任何受支持的包含FFMPEG音频数据的文件：*.wav，*.mp3甚至是视频文件，代码将自动从中提取音频

专家鉴别器的评估损失应降至约 0.25，Wav2Lip 评估同步损失应降至约 0.2，以获得良好的结果。