# Accelerate consumes more memory

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%3E%5C
**Category:** 🤗Accelerate\
**Created:** [November 18, 2025, 7:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588 "2025-11-18T07:28:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![DhineshR](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dhineshr/32/55576_2.png) [@DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDhineshR)%5C
**Post date:** [November 18, 2025, 7:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F1 "2025-11-18T07:28:06Z")

</div>

Hey Folks,

I’m trying to fine-tune qwen3-0.6B on gsm8k data using Trainer and Accelerate on a single GPU that has ~23GB memory. [Here](https://github.com/dhineshkumar-r/qwen3-0.6B-fine-tune/blob/main/fine-tune-gsm8k.py) is my training script.

When I launch the script with `python <script.py>`, the training works fine with ~20GB GPU memory utilization. However, when launching with `accelerate launch <script.py>`, I see OOM.

Following is my accelerate config.

```auto
compute_environment: LOCAL_MACHINE
debug: true
distributed_type: "NO"
downcast_bf16: 'no'
dynamo_config:
  dynamo_backend: INDUCTOR
enable_cpu_affinity: false
fsdp_config:
  fsdp_activation_checkpointing: false
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: false
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_version: 2
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

```

I don’t want to apply any memory optimization techniques such as gradient\_checkpoint or bf16 precision etc. The model is small and the batch\_size is set to very small number to work on one GPU. Am I missing anything in the configuration? Please let me know if you need more info. from my side.

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [November 18, 2025, 7:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F2 "2025-11-18T07:28:07Z")

</div>

Unlisted on November 18, 2025, 7:28am UTC



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [November 18, 2025, 8:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F3 "2025-11-18T08:48:40Z")

</div>

Listed on November 18, 2025, 8:48am UTC



---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 18, 2025, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F4 "2025-11-18T10:09:14Z")

</div>

[`dynamo_backend: INDUCTOR` might be culprit](https://huggingface.co/datasets/John6666/forum2/blob/main/accelerate_fsdp_oom_1.md)?

---

<div class="post-metadata">

**Author:** ![DhineshR](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dhineshr/32/55576_2.png) [@DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDhineshR)%5C
**Post date:** [November 19, 2025, 4:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F5 "2025-11-19T04:26:46Z")

</div>

It didn’t work. I tried removing `fsdp_config` as well. I still see OOM.

---

<div class="post-metadata">

**Author:** ![DhineshR](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dhineshr/32/55576_2.png) [@DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDhineshR)%5C
**Post date:** [November 19, 2025, 4:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F6 "2025-11-19T04:48:47Z")

</div>

I noticed multiple processes were spawned corresponding to number of GPUs on the machine when launching with accelerate even though `num_processes` is set to 1. During one of the gathering stage, OOM occurs.

Setting `distributed_type` to ‘MULTI\_GPU’ and `num_processes`= 1, works. However, I see a slight increase in GPU memory usage compared to python run, about 1GB difference.

When I set `dynamo_backend` to ‘INDUCTOR’ though, the memory usage is ~19GB, less than the above mentioned runs.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 19, 2025, 5:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F7 "2025-11-19T05:18:52Z")

</div>

Great!

If multiple GPUs are visible, using `accelerate launch` alone might cause strange behavior. If only a single GPU is visible, I don’t think there will be much difference…

* * *

Let’s unpack _why_ that “misconfigured multi-process → OOM during gather” situation almost certainly happened in your case.

I’ll go step by step:

1. How Accelerate decides how many processes to launch.
2. What “gather” / “all\_gather” actually does to memory.
3. Why “multiple processes per GPU” is especially bad at gather time.
4. How this lines up with what you observed.

* * *

## 1. How Accelerate decides how many processes to launch

### 1.1. `num_processes` is “how many worker processes to run”

Accelerate has two ways to decide process count:

- Command-line flag:  
`accelerate launch --num_processes N train.py`
- Config file:  
The YAML field `num_processes` that `accelerate config` writes.

If you _don’t_ specify `--num_processes`:

- The launcher typically uses the number of visible CUDA devices (one process per GPU) by default.
- Multiple processes can be launched **on the same machine** (but ideally not more than one per GPU).

So in a normal multi-GPU case:

- 4 GPUs → you run `accelerate launch --num_processes 4 ...` → 4 processes, each pinned to one GPU.

### 1.2. Your config said `num_processes: 1` but you still saw multiple processes

That tells us:

- The **effective** world size that Accelerate / `torch.distributed` saw was \> 1, even though the config file says 1.

- This mismatch is possible because:

So the most likely chain is:

- Some combination of:

- Result: more than one process ends up using the same GPU, even though your config says `num_processes: 1`.

From your description:

> multiple processes were spawned for each GPU on the machine

that’s exactly the kind of oversubscription the docs and examples warn against.

* * *

## 2. What “gather” / “all\_gather” actually does to memory

Accelerate’s `gather` operation is documented like this:

> The first dimension of the result is `num_processes` multiplied by the first dimension of the input tensors.

In other words, in a distributed setup:

- Each process has a tensor `x` of shape `[B, ...]`.
- `accelerator.gather(x)` (or an underlying all-gather) returns a tensor of shape:  
`[num_processes * B, ...]` **on each process**.

More generally, an **all-gather** pattern does this:

- Every GPU/process sends its local tensor to everyone else.
- After the operation, **every process** holds a concatenation of all local tensors.

So memory scaling is:

- Local tensor size: `S` bytes.
- `num_processes = P`.
- After all-gather, each process temporarily holds ~`P * S` bytes for that tensor.

That’s fine when:

- P is small (e.g. 2–4), and
- `S` is modest.

It becomes extremely expensive when:

- P is large (many processes), and
- `S` is large (e.g. logits of shape `[batch_size, seq_len, vocab_size]` or full hidden states).

People working with large models see exactly this: training is OK, but OOM appears **only** when an all-gather of results happens.

* * *

## 3. Why “multiple processes per GPU” is especially bad at gather time

Now combine the two ideas:

1. You have **more processes per GPU than you intended**.
2. `gather` / `all_gather` makes each process hold `P * S` worth of data.

What does that mean in practice for your GPU?

### 3.1. Memory during normal training steps

During a typical forward + backward:

- Each process on a GPU has:

If you accidentally have, say, 2 processes on a single GPU:

- You’ve roughly doubled the model copy / optimizer overhead.
- You’ve also doubled the activation and gradient memory (modulo some shared caching).

You might _just barely_ still fit, especially if your batch size is small.

### 3.2. Memory during the gather step

At the gather stage:

- Each process now allocates extra memory for the concatenated tensor from **all** processes:

- With 2 processes on one GPU, that GPU is now holding:

This is exactly the worst possible moment:

- Peak memory is highest when:

On a card where your “clean” setup already peaks around ~20GB, that extra “multiply by number of processes” factor at gather is enough to exceed 23GB and trigger OOM.

Research and engineering docs on all-gather emphasize that it is a bandwidth- and memory-heavy pattern; all GPUs end up holding the full result. For very large models, just the all-gather buffer can cause OOM by itself.

That’s exactly the pattern you saw: training runs until a gather step, then OOM.

* * *

## 4. How your “MULTI\_GPU + num\_processes=1” & Inductor tests fit into this

You then changed your config and saw:

1. `distributed_type: "MULTI_GPU", num_processes: 1`  
→ No more misguided multi-process per GPU; effectively world size = 1.  
→ No real all-gather; `gather` is trivial.  
→ OOM during gather disappears, but you pay ~1GB extra runtime overhead vs plain `python`.

2. Same as above, but `dynamo_backend: "INDUCTOR"`  
→ Inductor fuses kernels and changes activation layout.  
→ For your specific model + shapes, peak activation memory is lower, so the total VRAM drops to ~19GB.

These two observations confirm:

- The _primary_ reason for the original OOM was not the model itself; it was the **accidental extra processes per GPU** and the **all-gather at evaluation/metric time**.
- Once you force Accelerate into a clean single-process layout (`num_processes=1`) and avoid over-subscription, the OOM disappears even without changing model size or batch size.
- Inductor’s memory behavior then becomes an independent (and in your case, positive) factor.

* * *

## 5. Short conceptual summary

So, “Why did misconfigured multi-process → OOM during gather happen?”:

1. Accelerate (or your surrounding launcher) ended up running **multiple worker processes per GPU** , even though your YAML said `num_processes: 1`.

2. When the Trainer/Accelerate code hit a **gather** / **all-gather** (for evaluation/logging/metrics), each process tried to allocate a tensor containing **all processes’ outputs** , not just its own.

3. With multiple processes per GPU, that meant:

4. That temporary peak memory (during gather) exceeded your 23GB VRAM, causing OOM exactly at that stage.

Once you enforced **single process per GPU** (`distributed_type: MULTI_GPU`, `num_processes=1`, correct device visibility) and later used Inductor, the dangerous combination (multi-process + all-gather) went away, and the OOM stopped.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [November 19, 2025, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-consumes-more-memory%2F170588%2F8 "2025-11-19T17:19:01Z")

</div>

Closed on November 19, 2025, 5:19pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.