skyrim-vr-upscale

Foveated eye-buffer reconstruction for VR titles, loadable through kernels, built for and measured on Skyrim VR. The game renders each eye at reduced resolution; one kernel pass reconstructs the full-resolution buffer with quality that follows the lens: a 4x4 Catmull-Rom with an anti-ringing clamp and a fused unsharp term at the projection centre, plain bilinear at the rim, and a smoothstep-continuous blend of the two weight sets between, so quality falls off radially with no band seam anywhere in the image. Both eyes run in one launch with per-eye projection centres.

An A/B wipe between this kernel's foveated reconstruction and a plain bilinear upscale of the same reduced-resolution frame (media/make_hero.py). Full Catmull-Rom quality holds inside the circle over the statue and falls off smoothly to bilinear at the rim, with no band seam. The frame is a stone dragon path-traced with phanerozoic/pathtracer-diff, standing in for a VR eye buffer; it is a reconstruction demonstration.

The open VR injection ecosystem upscales in two tiers with a hard edge between them (vrperfkit runs FSR1 inside a radius and bilinear outside, per 16x16 workgroup), and the visible ring at that edge is the known artifact of the design. This kernel replaces the two-tier switch with a continuous filter family, and adds the determinism discipline of a compute kernel: reconstruction is bitwise reproducible across runs, launch configurations, and CUDA architectures.

Usage

import torch
from kernels import get_kernel

svr = get_kernel("phanerozoic/skyrim-vr-upscale", version=1,
                 trust_remote_code=True)

lo = torch.empty(2, 1725, 1552, 4, dtype=torch.uint8, device="cuda")  # eyes
hi = svr.upscale(lo, out_hw=(2240, 2016),
                 center=[[0.46, 0.52], [0.54, 0.52]],   # per-eye, fractional
                 radii=(0.30, 0.70), sharp=0.4)

src is [H, W, 4] or [B, H, W, 4] RGBA, uint8 or float32. radii are fractions of the output height: full quality inside r0, bilinear outside r1. center defaults to the image centre; a VR runtime's real per-eye projection centres go straight through. version selects the release branch; trust_remote_code is required by kernels for publishers without the trusted-publisher mark.

How it works

Each output pixel maps to its source coordinate under the standard half-pixel convention and evaluates one 4x4 filter whose weights are a radial blend w = cr + t * (tent - cr), with t = smoothstep(r0, r1, r) in eccentricity from the projection centre. Accumulation is centre plus weighted differences, out = c + sum w_i (s_i - c): the weights sum to one analytically, so a constant field reproduces exactly regardless of floating-point weight error. Catmull-Rom overshoot at edges is confined to the inner 2x2 hull (the bilinear part already lies inside it), and a fused unsharp term out += sharp * (1 - t) * (out - bilinear) restores acutance in the fovea, fading with the same radial profile as the filter. Beyond r1 the kernel takes a 4-tap fast path that is arithmetically the t = 1 limit of the blended path, so the fast-path boundary introduces no seam of its own.

Every floating-point step is an explicit round-per-op intrinsic (__fadd_rn / __fmul_rn / __fmaf_rn / __fsqrt_rn), so ptxas cannot contract expressions differently per target and output bytes are a function of the code, not the architecture. One thread owns one output pixel and writes it once: no atomics, bitwise deterministic.

Correctness

The suite is analytic-first; every property below is a test.

• Constant fields are bit-exact in every band, both dtypes, with sharpening on: the centre-difference form makes every contribution identically zero.
• Same-size is a bitwise pass-through (sharp = 0): Catmull-Rom is interpolating and both weight sets are the delta at phase zero, at any radii.
• Linear ramps reconstruct as themselves at every eccentricity, away from the clamped border: Catmull-Rom, tent, their blend, and the unsharp term all have linear precision.
• The periphery is bilinear: with radii = (0, 0) the kernel matches torch.nn.functional.interpolate(mode="bilinear") to float rounding.
• No ringing: a step edge reconstructs inside the local hull; the output range is the input range.
• The float64 reference agrees: a vectorized f64 model of the whole pipeline bounds the f32 kernel at 2e-5 max error on detailed content.
• Batched equals singles, bitwise, with distinct per-eye centres.
• Deterministic, bitwise, on repeat calls.
• Golden digests: SHA-256 of reconstructions of hash-built frames, minted on sm89. Digest equality on another card certifies bit-identical output across CUDA architectures, torch releases, and toolchains.
• Foveal quality: on band-limited detailed content, foveal MSE against ground truth is under 0.7x bilinear's, while peripheral MSE matches bilinear within 5%.

Kernel throughput

RTX 6000 Ada (sm89), both eyes per launch, medians of cudaEvent timings with buffers resident (bench/eye_bench.py). Output shapes span SteamVR render targets for a Quest 3 over Link; sources are the renderScale = 0.77 buffers an injector feeds. Baselines on the same tensors: the kernel pinned fully bilinear (radii = (0, 0)), pinned fully foveal (radii = (9, 9.9)), and torch's own resamplers.

At the standard shape the full graded pass costs 0.10 ms per eye in uint8: about 0.9% of a 90 Hz frame budget, faster than torch's uniform bilinear resample of the same buffers, and 4.6x faster than torch bicubic, while delivering Catmull-Rom quality in the fovea. The graded frame prices at its foveal ceiling (0.202 vs 0.203 ms): the radial blend itself is free, and the periphery fast path buys headroom that widens as r1 tightens.

Frame-rate benefit in VR

An upscaler earns frame time by rendering fewer, cheaper pixels, then reconstructing full resolution. Measured on an RTX 6000 Ada with GPU timestamp queries, native D3D11 path (bench/perf_hlsl.cpp), both eyes per frame, on a demanding scene (Quest 3 high, 2464x2736/eye):

Reconstruction costs about 0.23 ms per frame at this resolution regardless of scene, roughly 2% of a 90 Hz budget, which is why the measured speedup sits just under the pixel-count ceiling: the scene render drops by the full pixel ratio and only the reconstruction is added back. The gain scales with how GPU-bound the scene is (the number above is a heavy shading load); a CPU-bound scene gains nothing from any resolution upscaler. renderScale is the dial between frame rate and source detail, and the foveal reconstruction buys some of that detail back.

The CUDA-bridge path (bench/perf_cuda_bridge.cpp) measures 1.05x at renderScale 0.77 on the same scene: the CUDA-D3D11 interop map/unmap and staging copies add about 0.4 ms per frame on top of a 0.1 ms kernel, which is why the in-game path uses the HLSL shader.

In-game injection

The kernel drives an injection path for OpenVR/SteamVR titles, not Skyrim alone. A vrperfkit-derived dxgi wrapper hooks the game's D3D11 device and its compositor submit, reads the runtime's per-eye projection centres, and reconstructs each submitted eye buffer. Any OpenVR D3D11 title on SteamVR hits the same path; this was verified headless on SteamVR's null driver with a non-game OpenVR harness (bench/openvr_inject_harness.cpp), where the injector hooked and ran the kernel on submitted eye textures with no headset or game present.

game/ carries the injection artifacts: svr_upscale.hlsl, a D3D11 compute-shader port of the kernel that reconstructs bit-for-bit what the CUDA kernel does (within 1/255, verified by bench/hlsl_vs_cuda_check.cpp) and runs entirely in D3D11 for the numbers above; svr_bridge.cu, which runs the published CUDA kernel in-process through cudaGraphicsD3D11RegisterResource; and the vrperfkit integration. The .cu kernel is torch-free behind a C ABI (svr_cuda/upscale_launch.h) so the Hub build and the in-process bridge compile the identical source. Fixed-foveation radii follow the headset lens, not eye tracking.

Requirements and limits

• NVIDIA GPU, compute capability 7.5+; CUDA.
• RGBA, uint8 or float32, contiguous [B, H, W, 4]; sources at least 2x2. Any output size; no alignment constraints.
• Fixed foveation: the centre is per-eye but static per call. sRGB filtering is performed in the stored space, matching the injection ecosystem's practice.
• Published variants are Linux x86_64. On Windows, or for a torch build without a matching variant, load_local.py JIT-builds the same source and exposes the identical API.

References

Catmull and Rom, "A class of local interpolating splines" (1974); Mitchell and Netravali, "Reconstruction filters in computer graphics" (SIGGRAPH 1988); Guenter et al., "Foveated 3D graphics" (SIGGRAPH Asia 2012); Patney et al., "Towards foveated rendering for gaze-tracked virtual reality" (SIGGRAPH Asia 2016); fholger/vrperfkit (the two-tier radial upscale this kernel supersedes); AMD FidelityFX Super Resolution 1.0 and NVIDIA Image Scaling (the uniform single-pass upscalers of the same ecosystem).

License

Apache-2.0.