I tested coordinate and Morton ordering in a WCSPH proxy application for my MSc thesis at NTNU. Both brought interacting particles closer together in memory. The solver still didn’t get a stable end-to-end speedup.
WCSPH, or weakly compressible smoothed-particle hydrodynamics, represents fluids using particles that interact with nearby particles within a limited support radius. It’s useful for free surfaces and moving boundaries. For the implementation, that means irregular memory access and repeated neighborhood traversal. I investigated its performance on an NVIDIA H100 multi-GPU node.
Sorting particles
The locality measurements looked promising. In the scale-4, eight-GPU runs, Morton ordering with sorting intervals of 20 and 100 placed about 78% of accepted pairs within 127 particle indices of each other at the latest sample. This index distance was my proxy for locality; it didn’t measure the complete timestep’s performance.
The implementation sorts full particle records, which costs time. Synchronization and other timestep work also contribute to runtime. In repeated scale-4 runs, the measured Morton configurations were slower on average despite the improved locality proxy.
Testing NCCL
I also compared communication using NCCL and host-staged MPI. NCCL microbenchmarks showed a higher device-resident communication ceiling. In the final paired solver runs, though, the tested NCCL backend was slower than MPI.
Those results are specific to the implementation and runs I tested. The full exchange path includes costs beyond the transfer itself, so changing the backend doesn’t guarantee a faster exchange.
I instrumented the proxy to separate CUDA timestep phases, neighbor search, pair construction, particle migration, and border exchange. Host-side particle handling, rank imbalance, and barrier waiting all mattered too. Particle count alone wasn’t enough to describe each rank’s workload.
This is the lesson I’ll take into future performance work: measure the complete path before calling an optimization a win. Locality measurements and communication microbenchmarks helped explain individual costs, but I needed the solver measurements to see whether the changes actually reduced runtime.
From my thesis, “Performance Modeling of a WCSPH Proxy Application on Multi-GPU Systems”, supervised by Jan Christian Meyer. Submitted and awaiting grading.