把传统sd训练lora的脚本看得七七八八了，记录一下。

大体就是通过要训练的模型去新建一个network,network里面包含着替换掉原模型某些层的lora module,然后通过network.apply to成功替换到到要训练的模型上，这样在原模型的训练过程中，lora的参数也回随之更新。

network.apply_to(text_encoder, unet, train_text_encoder, train_unet)

更根本的就是lora module在原有块的forward上，加入了lora参数矩阵相乘或其他操作生成的结果，来进行模型的微调。但根据替换的不同块，怎么设置lora也不同，还是要具体模型具体分析的。

下面先来看看传统的sd lora训练中替换了unet的哪些模块

down_blocks.0.attentions.0 Transformer2DModel
down_blocks.0.attentions.1 Transformer2DModel
down_blocks.0.resnets.0 ResnetBlock2D
down_blocks.0.resnets.1 ResnetBlock2D
down_blocks.0.downsamplers.0 Downsample2D
down_blocks.1.attentions.0 Transformer2DModel
down_blocks.1.attentions.1 Transformer2DModel
down_blocks.1.resnets.0 ResnetBlock2D
down_blocks.1.resnets.1 ResnetBlock2D
down_blocks.1.downsamplers.0 Downsample2D
down_blocks.2.attentions.0 Transformer2DModel
down_blocks.2.attentions.1 Transformer2DModel
down_blocks.2.resnets.0 ResnetBlock2D
down_blocks.2.resnets.1 ResnetBlock2D
down_blocks.2.downsamplers.0 Downsample2D
down_blocks.3.resnets.0 ResnetBlock2D
down_blocks.3.resnets.1 ResnetBlock2D
up_blocks.0.resnets.0 ResnetBlock2D
up_blocks.0.resnets.1 ResnetBlock2D
up_blocks.0.resnets.2 ResnetBlock2D
up_blocks.0.upsamplers.0 Upsample2D
up_blocks.1.attentions.0 Transformer2DModel
up_blocks.1.attentions.1 Transformer2DModel
up_blocks.1.attentions.2 Transformer2DModel
up_blocks.1.resnets.0 ResnetBlock2D
up_blocks.1.resnets.1 ResnetBlock2D
up_blocks.1.resnets.2 ResnetBlock2D
up_blocks.1.upsamplers.0 Upsample2D
up_blocks.2.attentions.0 Transformer2DModel
up_blocks.2.attentions.1 Transformer2DModel
up_blocks.2.attentions.2 Transformer2DModel
up_blocks.2.resnets.0 ResnetBlock2D
up_blocks.2.resnets.1 ResnetBlock2D
up_blocks.2.resnets.2 ResnetBlock2D
up_blocks.2.upsamplers.0 Upsample2D
up_blocks.3.attentions.0 Transformer2DModel
up_blocks.3.attentions.1 Transformer2DModel
up_blocks.3.attentions.2 Transformer2DModel
up_blocks.3.resnets.0 ResnetBlock2D
up_blocks.3.resnets.1 ResnetBlock2D
up_blocks.3.resnets.2 ResnetBlock2D
mid_block.attentions.0 Transformer2DModel
mid_block.resnets.0 ResnetBlock2D
mid_block.resnets.1 ResnetBlock2D

大概如上，diffusers的unet好像再次封装过，一般downsample和upsample是12层，这边变成了四块，然后后面再到这些块的子模块去替换底层的模块。

刚去打印了一下这些快拿下，具体哪些又被替换：

up_blocks.0.resnets.1 ResnetBlock2D
not in Linear or Conv2d: ResnetBlock2D
not in Linear or Conv2d: norm1 GroupNorm
in name: conv1 Conv2d
in name: time_emb_proj Linear
not in Linear or Conv2d: norm2 GroupNorm
not in Linear or Conv2d: dropout Dropout
in name: conv2 Conv2d
not in Linear or Conv2d: nonlinearity SiLU
in name: conv_shortcut Conv2d
up_blocks.0.resnets.2 ResnetBlock2D
not in Linear or Conv2d: ResnetBlock2D
not in Linear or Conv2d: norm1 GroupNorm
in name: conv1 Conv2d
in name: time_emb_proj Linear
not in Linear or Conv2d: norm2 GroupNorm
not in Linear or Conv2d: dropout Dropout
in name: conv2 Conv2d
not in Linear or Conv2d: nonlinearity SiLU
in name: conv_shortcut Conv2d
up_blocks.0.upsamplers.0 Upsample2D
not in Linear or Conv2d: Upsample2D
in name: conv Conv2d
up_blocks.1.attentions.0 Transformer2DModel
not in Linear or Conv2d: Transformer2DModel
not in Linear or Conv2d: norm GroupNorm
in name: proj_in Conv2d
not in Linear or Conv2d: transformer_blocks ModuleList
not in Linear or Conv2d: transformer_blocks.0 BasicTransformerBlock
not in Linear or Conv2d: transformer_blocks.0.norm1 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.attn1 Attention
in name: transformer_blocks.0.attn1.to_q Linear
in name: transformer_blocks.0.attn1.to_k Linear
in name: transformer_blocks.0.attn1.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out ModuleList
in name: transformer_blocks.0.attn1.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.norm2 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.attn2 Attention
in name: transformer_blocks.0.attn2.to_q Linear
in name: transformer_blocks.0.attn2.to_k Linear
in name: transformer_blocks.0.attn2.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out ModuleList
in name: transformer_blocks.0.attn2.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.norm3 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.ff FeedForward
not in Linear or Conv2d: transformer_blocks.0.ff.net ModuleList
not in Linear or Conv2d: transformer_blocks.0.ff.net.0 GEGLU
in name: transformer_blocks.0.ff.net.0.proj Linear
not in Linear or Conv2d: transformer_blocks.0.ff.net.1 Dropout
in name: transformer_blocks.0.ff.net.2 Linear
in name: proj_out Conv2d
up_blocks.1.attentions.1 Transformer2DModel
not in Linear or Conv2d: Transformer2DModel
not in Linear or Conv2d: norm GroupNorm
in name: proj_in Conv2d
not in Linear or Conv2d: transformer_blocks ModuleList
not in Linear or Conv2d: transformer_blocks.0 BasicTransformerBlock
not in Linear or Conv2d: transformer_blocks.0.norm1 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.attn1 Attention
in name: transformer_blocks.0.attn1.to_q Linear
in name: transformer_blocks.0.attn1.to_k Linear
in name: transformer_blocks.0.attn1.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out ModuleList
in name: transformer_blocks.0.attn1.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.norm2 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.attn2 Attention
in name: transformer_blocks.0.attn2.to_q Linear
in name: transformer_blocks.0.attn2.to_k Linear
in name: transformer_blocks.0.attn2.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out ModuleList
in name: transformer_blocks.0.attn2.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.norm3 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.ff FeedForward
not in Linear or Conv2d: transformer_blocks.0.ff.net ModuleList
not in Linear or Conv2d: transformer_blocks.0.ff.net.0 GEGLU
in name: transformer_blocks.0.ff.net.0.proj Linear
not in Linear or Conv2d: transformer_blocks.0.ff.net.1 Dropout
in name: transformer_blocks.0.ff.net.2 Linear
in name: proj_out Conv2d

像上面not in Linear or Conv2d就是没有被lora替换的块，蛮多都是ModuleList，GroupNorm，SiLU等类型的，大概要么就是比较大的模块，要么就是归一层和激活层不用训练。

看了上面这些层，连忙回去改了一下，这下子DynamiCrafter要进行训练的层应该就能确定了,如下

diffusion_model.input_blocks.1.0 ResBlock
not in Linear or Conv2d:  ResBlock
not in Linear or Conv2d: in_layers Sequential
not in Linear or Conv2d: in_layers.0 GroupNormSpecific
not in Linear or Conv2d: in_layers.1 SiLU
in name: in_layers.2 Conv2d
not in Linear or Conv2d: h_upd Identity
not in Linear or Conv2d: emb_layers Sequential
not in Linear or Conv2d: emb_layers.0 SiLU
in name: emb_layers.1 Linear
not in Linear or Conv2d: out_layers Sequential
not in Linear or Conv2d: out_layers.0 GroupNormSpecific
not in Linear or Conv2d: out_layers.1 SiLU
not in Linear or Conv2d: out_layers.2 Dropout
in name: out_layers.3 Conv2d
not in Linear or Conv2d: skip_connection Identity
not in Linear or Conv2d: temopral_conv TemporalConvBlock
not in Linear or Conv2d: temopral_conv.conv1 Sequential
not in Linear or Conv2d: temopral_conv.conv1.0 GroupNorm
not in Linear or Conv2d: temopral_conv.conv1.1 SiLU
in name: temopral_conv.conv1.2 Conv3d
not in Linear or Conv2d: temopral_conv.conv2 Sequential
not in Linear or Conv2d: temopral_conv.conv2.0 GroupNorm
not in Linear or Conv2d: temopral_conv.conv2.1 SiLU
not in Linear or Conv2d: temopral_conv.conv2.2 Dropout
in name: temopral_conv.conv2.3 Conv3d
not in Linear or Conv2d: temopral_conv.conv3 Sequential
not in Linear or Conv2d: temopral_conv.conv3.0 GroupNorm
not in Linear or Conv2d: temopral_conv.conv3.1 SiLU
not in Linear or Conv2d: temopral_conv.conv3.2 Dropout
in name: temopral_conv.conv3.3 Conv3d
not in Linear or Conv2d: temopral_conv.conv4 Sequential
not in Linear or Conv2d: temopral_conv.conv4.0 GroupNorm
not in Linear or Conv2d: temopral_conv.conv4.1 SiLU
not in Linear or Conv2d: temopral_conv.conv4.2 Dropout
in name: temopral_conv.conv4.3 Conv3d
diffusion_model.input_blocks.1.0.temopral_conv TemporalConvBlock
not in Linear or Conv2d:  TemporalConvBlock
not in Linear or Conv2d: conv1 Sequential
not in Linear or Conv2d: conv1.0 GroupNorm
not in Linear or Conv2d: conv1.1 SiLU
in name: conv1.2 Conv3d
not in Linear or Conv2d: conv2 Sequential
not in Linear or Conv2d: conv2.0 GroupNorm
not in Linear or Conv2d: conv2.1 SiLU
not in Linear or Conv2d: conv2.2 Dropout
in name: conv2.3 Conv3d
not in Linear or Conv2d: conv3 Sequential
not in Linear or Conv2d: conv3.0 GroupNorm
not in Linear or Conv2d: conv3.1 SiLU
not in Linear or Conv2d: conv3.2 Dropout
in name: conv3.3 Conv3d
not in Linear or Conv2d: conv4 Sequential
not in Linear or Conv2d: conv4.0 GroupNorm
not in Linear or Conv2d: conv4.1 SiLU
not in Linear or Conv2d: conv4.2 Dropout
in name: conv4.3 Conv3d
diffusion_model.input_blocks.1.1 SpatialTransformer
not in Linear or Conv2d:  SpatialTransformer
not in Linear or Conv2d: norm GroupNorm
in name: proj_in Linear
not in Linear or Conv2d: transformer_blocks ModuleList
not in Linear or Conv2d: transformer_blocks.0 BasicTransformerBlock
not in Linear or Conv2d: transformer_blocks.0.attn1 CrossAttention
in name: transformer_blocks.0.attn1.to_q Linear
in name: transformer_blocks.0.attn1.to_k Linear
in name: transformer_blocks.0.attn1.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out Sequential
in name: transformer_blocks.0.attn1.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.ff FeedForward
not in Linear or Conv2d: transformer_blocks.0.ff.net Sequential
not in Linear or Conv2d: transformer_blocks.0.ff.net.0 GEGLU
in name: transformer_blocks.0.ff.net.0.proj Linear
not in Linear or Conv2d: transformer_blocks.0.ff.net.1 Dropout
in name: transformer_blocks.0.ff.net.2 Linear
not in Linear or Conv2d: transformer_blocks.0.attn2 CrossAttention
in name: transformer_blocks.0.attn2.to_q Linear
in name: transformer_blocks.0.attn2.to_k Linear
in name: transformer_blocks.0.attn2.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out Sequential
in name: transformer_blocks.0.attn2.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out.1 Dropout
in name: transformer_blocks.0.attn2.to_k_ip Linear
in name: transformer_blocks.0.attn2.to_v_ip Linear
not in Linear or Conv2d: transformer_blocks.0.norm1 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.norm2 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.norm3 LayerNorm
in name: proj_out Linear
diffusion_model.input_blocks.1.2 TemporalTransformer
not in Linear or Conv2d:  TemporalTransformer
not in Linear or Conv2d: norm GroupNorm
in name: proj_in Linear
not in Linear or Conv2d: transformer_blocks ModuleList
not in Linear or Conv2d: transformer_blocks.0 BasicTransformerBlock
not in Linear or Conv2d: transformer_blocks.0.attn1 CrossAttention
in name: transformer_blocks.0.attn1.to_q Linear
in name: transformer_blocks.0.attn1.to_k Linear
in name: transformer_blocks.0.attn1.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out Sequential
in name: transformer_blocks.0.attn1.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn1.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.ff FeedForward
not in Linear or Conv2d: transformer_blocks.0.ff.net Sequential
not in Linear or Conv2d: transformer_blocks.0.ff.net.0 GEGLU
in name: transformer_blocks.0.ff.net.0.proj Linear
not in Linear or Conv2d: transformer_blocks.0.ff.net.1 Dropout
in name: transformer_blocks.0.ff.net.2 Linear
not in Linear or Conv2d: transformer_blocks.0.attn2 CrossAttention
in name: transformer_blocks.0.attn2.to_q Linear
in name: transformer_blocks.0.attn2.to_k Linear
in name: transformer_blocks.0.attn2.to_v Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out Sequential
in name: transformer_blocks.0.attn2.to_out.0 Linear
not in Linear or Conv2d: transformer_blocks.0.attn2.to_out.1 Dropout
not in Linear or Conv2d: transformer_blocks.0.norm1 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.norm2 LayerNorm
not in Linear or Conv2d: transformer_blocks.0.norm3 LayerNorm
in name: proj_out Linear

我节选了一部分，这么一看和上面就对齐多了，差不多就是这样三个input_blocks组成diffusers封装的down_block，上面看上去是长了点，很多都是不用训练的，我打出来确认一下，此外，dynamiCrafter多了Conv3d要训练，应该是要加上去的，我待会儿再用作者给的正则去匹配一下看看。

唔，匹配下来，还是比较一致的，似乎就groupNorm在上面没出现，但我看传统的sd lora里也没训练groupNorm,问题应该不大。

model.diffusion_model.output_blocks.5.0.temopral_conv.conv1.0.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv1.0.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv1.2.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv1.2.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv2.0.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv2.0.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv2.3.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv2.3.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv3.0.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv3.0.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv3.3.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv3.3.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv4.0.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv4.0.bias
model.diffusion_model.output_blocks.5.0.temopral_conv.conv4.3.weight
model.diffusion_model.output_blocks.5.0.temopral_conv.conv4.3.bias
model.diffusion_model.output_blocks.5.2.norm.weight
model.diffusion_model.output_blocks.5.2.norm.bias
model.diffusion_model.output_blocks.5.2.proj_in.weight
model.diffusion_model.output_blocks.5.2.proj_in.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn1.to_q.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn1.to_k.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn1.to_v.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn1.to_out.0.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn1.to_out.0.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.ff.net.0.proj.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.ff.net.0.proj.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.ff.net.2.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.ff.net.2.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn2.to_q.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn2.to_k.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn2.to_v.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn2.to_out.0.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.attn2.to_out.0.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.norm1.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.norm1.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.norm2.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.norm2.bias
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.norm3.weight
model.diffusion_model.output_blocks.5.2.transformer_blocks.0.norm3.bias
model.diffusion_model.output_blocks.5.2.proj_out.weight
model.diffusion_model.output_blocks.5.2.proj_out.bias

那么确认要替换的模块就准备的差不多了，下一话，就要来看看具体怎么去替换成lora的Module了，linear和Conv2d可能可以直接用，Conv3d要再看看，以及原脚本里还有对这个is_conv2d_1x1的判断，一级一些skip module。