# How to load pretrained model with custom model layers

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-pretrained-model-with-custom-model-layers%2F26425%3E%5C
**Category:** Beginners\
**Created:** [November 21, 2022, 4:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-pretrained-model-with-custom-model-layers%2F26425 "2022-11-21T04:52:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![icedpanda](https://avatars.discourse-cdn.com/v4/letter/i/df705f/32.png) [@icedpanda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ficedpanda)%5C
**Post date:** [November 21, 2022, 4:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-pretrained-model-with-custom-model-layers%2F26425%2F1 "2022-11-21T04:52:06Z")

</div>

Hi

How can I load pre-trained BART model weights into a custom model layer?

I have a custom decoder layer with addition `nn.Module` but pretrain BART model like `facebook/bart-base` has prefix `decoder.layer.` keys cant match the custom layer.

```python
from transformers.models.bart.modeling_bart import BartEncoder, BartDecoder

class BartDecoder(BartPretrainedModel):
  def __init__ (self, config: BartConfig, embed_tokens: Optional[nn.Embedding] = None):
        super(). __init__ (config)
        # assume here are same 
        # ...

class BartDecoderLayer(nn.Module):
    def __init__ (self, config: BartConfig):
        super(). __init__ ()
        # assume we have original modules here 
        # ...
        self.some_new_modules1 = something_new
        self.some_new_modules2 = something_new

```

How can I load pretrained `facebook/bart-base` into this decoder and leave the rest of `self.some_new_modules1` and `self.some_new_modules2`? The `facebook/bart-base` has prefix keys like `endoer.layer..` and `decoder.layer..`

Thanks

---

<div class="post-metadata">

**Author:** ![Ericosmic](https://avatars.discourse-cdn.com/v4/letter/e/b9bd4f/32.png) [@Ericosmic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEricosmic)%5C
**Post date:** [September 6, 2023, 12:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-pretrained-model-with-custom-model-layers%2F26425%2F2 "2023-09-06T12:36:23Z")

</div>

I have same request. Has anyone could give an answer.

---

<div class="post-metadata">

**Author:** ![voorhs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voorhs/32/18997_2.png) [@voorhs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvoorhs)%5C
**Post date:** [September 12, 2023, 4:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-pretrained-model-with-custom-model-layers%2F26425%2F3 "2023-09-12T16:30:02Z")

</div>

I think the easiest way is following:

- follow the source code structure of pre-trained model
- name all attributes containing pre-trained modules the same as in the pre-trained model
- code a `PreTrainedModel` class and add the `base_model_prefix` attribute to it with the name of the pre-trained model, for example `"bert"` or `"mpnet"`.

An example might be my small project.: [GitHub - voorhs/hssa](https://github.com/voorhs/hssa)
