# Model (Pipeline) Parallelism in SLURM cluster

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-pipeline-parallelism-in-slurm-cluster%2F68138%3E%5C
**Category:** DeepSpeed\
**Created:** [January 6, 2024, 8:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-pipeline-parallelism-in-slurm-cluster%2F68138 "2024-01-06T08:23:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![arindammajee](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@arindammajee](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farindammajee)%5C
**Post date:** [January 6, 2024, 8:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-pipeline-parallelism-in-slurm-cluster%2F68138%2F1 "2024-01-06T08:23:09Z")

</div>

I have a SLURM based cluster with 2 nodes each having a single NVIDIA RTX 4090 GPU (24 GB). Hence, total 2 GPU with 2\*24=48 GB memory. Can I load and fine-tune large LLMs in this cluster? If yes, then how? Any kind of help or clue would be appreciated. Thanks in advance.
