Our Space whisperx-transcription is consistently failing to start with the same container-runtime error, across three separate attempts today (two Factory Rebuilds and one normal rebuild triggered by a commit):
Exit code: 128. Reason: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT_ATTR_IDMAP on /usr/bin/nvidia-cuda-mps-control: invalid argument (maybe the filesystem used doesn’t support idmap mounts on this kernel?)
This fails at container init, before our application code runs, so it doesn’t appear to be anything in our Dockerfile or app. We’ve already ruled out our own code as the cause:
Build phase always completes successfully.
The failure happens identically at container start every time.
We’ve tried: Factory Rebuild (twice) and a fresh commit-triggered rebuild — all three attempts hit the exact same error
I have no idea what could be the problem and how to fix it
The factory rebuilds are a strong hint this might not be your code — and there’s a sibling thread hitting the identical error (“All paid GPU pools fail to start: MOUNT_ATTR_IDMAP error at container init”), which makes me think this could be host-side rather than yours.
My read on what’s failing: MOUNT_ATTR_IDMAP is the kernel’s idmapped-mount mechanism (UID/GID remapping for container isolation). nvidia-cuda-mps-control gets injected by the GPU runtime at container start, not something from your image — and “invalid argument” on an idmap mount usually means the host’s kernel/filesystem can’t honor the request. I can’t see how anything in a Dockerfile could produce this error, and it happens before your app code ever runs, which matches everything you observed.
Worth trying:
A different GPU pool — if the Space boots there, that would isolate it to one pool’s node config, and that’s your fastest unblock.
If it’s broken on every pool, it could be HF’s fleet: worth opening with the pool name and this exact error string. The fix would live in their node image, not yours.
I’d stop rebuilding. Three identical failures across factory rebuilds is the tell — once it looks host-side, more rebuilds just spend the same GPU minutes on the same host config.
Your build succeeding every time is the evidence that matters most here. The container never got far enough to touch your code, as far as I can tell.
Hand this to your LLM with one rule: forget what things are called, trace only what physically happens, in order. The answer is in the motion, never in the names.