Repository navigation
feat(lcao): add optional LibRI GPU contractions through DDLA - #8169
Stardust0831 wants to merge 29 commits into
Conversation
|
the LibDDLA version which is compatible with LibRI GPU is https://github.com/userchba/LibDDLA/releases/tag/v0.0.4, current LibRI GPU can't correctly compute symmetry situation which LibRI CPU can compute correctly |
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
Static review of f07d8d14f39d26782490cc4bc15d0a58c5700e62 with independent source review and adjudication. No build/runtime/MPI/GPU tests were run; CI diagnostics were unavailable.
| (default OFF), `USE_CUDA=ON`, and a CUDA build of DDLA. Like other dependent | ||
| options, GPU RI is disabled when its LibRI/CUDA prerequisites are disabled. | ||
| INPUT `exx_device` defaults to `cpu`, retaining CPU contractions even with | ||
| `device gpu`. To use GPU contractions, set both `device gpu` and `exx_device gpu`. |
There was a problem hiding this comment.
[P2] State the GPU visibility requirement for MPI runs
For two ranks on one node, exposing one different GPU per rank through CUDA_VISIBLE_DEVICES passes ABACUS device setup but makes the pinned LibRI validate_local_gpu_count() throw at the first GPU contraction (local_size=2 > device_count=1). This backend therefore requires each process to see at least as many GPUs as there are node-local ranks, a constraint missing here. Please document it with a compatible launch layout, or extend the compatibility patch to respect the existing per-rank binding.
There was a problem hiding this comment.
The guide and device_exx help now require a common visible GPU list with at least one GPU per node-local rank, and the guide includes a compatible two-rank launch. This addresses the documentation request; per-rank single-GPU masking remains outside the supported layout.
| Exx_Info& exx_info) | ||
| { | ||
| #ifdef __EXX | ||
| const bool use_gpu = inp.exx_device == "gpu"; |
There was a problem hiding this comment.
Non-blocking: Please clarify the selector's scope for independent LR/RDMFT exchange engines. Standalone LR, fresh LR-kernel setup after KS, and RDMFT construct their own Exx_LRI objects, which retain CPU mode despite accepted exx_device=gpu; matching-kernel KS pointer reuse retains GPU mode. If those engines are in scope, pass the selection to them; otherwise document or reject the unsupported combinations. A follow-up after merge is fine for these additional workflows.
There was a problem hiding this comment.
Non-blocking: current share_exx_lri() creates a fresh CPU workspace and shares geometry only, so the GPU-inheritance sentence in device_exx help, the installation guide and the PR body is stale. Please state that LR remains on CPU even when it reuses KS geometry.
| { | ||
| return 1; | ||
| } | ||
| lri.cal_mode = RI::LRI_Cal_Mode::GPU; |
There was a problem hiding this comment.
Non-blocking: Consider an opt-in numerical regression for the patch's signed accumulation in a follow-up after merge. This consumer switches cal_mode but then executes only CPU nrm2, and the configure probe only compiles/links. Alongside the reported force/stress runs, a tiny real/complex CPU-versus-GPU case with nonempty output and repeated +1/-1 contractions would preserve the specific accumulation fix across dependency updates.
There was a problem hiding this comment.
The new opt-in libri_gpu_accumulation test checks all 18 contractions for real and complex tensors using a nonzero copied seed and repeated -1/+1 accumulation against the CPU result. This addresses the requested committed regression coverage. It is enabled with ENABLE_LIBRI_GPU_TESTS=ON.
|
To be consistent, I think device_exx maybe better than exx_device |
Thank you for the review and suggestions; I’ve renamed exx_device to device_exx. |
|
@Growl1234 Could you check the cmake changes for this PR? |
| if(ENABLE_LIBRI_GPU) | ||
| message( | ||
| FATAL_ERROR | ||
| "ENABLE_LIBRI_GPU requires an explicitly supplied, patched LIBRI_DIR; " |
There was a problem hiding this comment.
Does it mean that LibRI master branch hasn't yet accepted the LibRI changes? If this is true then this PR should not be merged on abacus side first.
There was a problem hiding this comment.
LibDDLA has CMake config export, why is FindDDLA module needed here?
|
I have no further comment within this PR's scope. |
|
An update on the LibRI dependency fixes:
The ABACUS test results reported in this PR still use the temporary compatibility patch. The new direct-accumulation implementation has passed LibRI-level GPU regression tests with DDLA v0.0.4, but this PR has not yet been updated to use it or retested with full ABACUS HSE calculations. I plan to move this PR back to draft while the remaining LibRI fix is pending upstream. Once merged, I will update the dependency, remove the compatibility patch, address the remaining review comments, and sync with develop. I will then rerun the HSE single-point energy, force, and stress regressions before marking this PR ready for review again. |
ABACUS-PR-BOT
left a comment
There was a problem hiding this comment.
The GPU-enabled main test target MODULE_RI_exx_workspace needs its DDLA link dependency; the inline comments also cover two nonblocking follow-ups. This review is source-only: no build or test was run, author-reported results remain attributed, and CI was not independently checked.
| foreach( | ||
| _abacus_feature_target IN | ||
| ITEMS nlohmann_json::nlohmann_json | ||
| LibRI::LibRI |
There was a problem hiding this comment.
[P2] Carry the DDLA link dependency into the RI workspace test
With BUILD_TESTING=ON, ENABLE_LIBRI_GPU=ON and CMAKE_BUILD_TYPE=Debug, these copied requirements enable __GPU_RI in MODULE_RI_exx_workspace, whose link list has no LibRI::LibRI or DDLA dependency. Its cal_Hs()/cal_force() calls instantiate runtime GPU dispatch even when the objects default to CPU, leaving DDLA references unresolved in an unoptimized link. Please link this target to LibRI::LibRI, or supply a consistent CPU-only compile interface, and verify that specific main-project target.
| { | ||
| // This standalone dependency test has no ABACUS runtime to initialize MPI. | ||
| int provided = 0; | ||
| if (MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided) != MPI_SUCCESS) |
There was a problem hiding this comment.
Non-blocking: the standalone-test rationale explains these direct MPI_Init_thread/MPI_Finalize calls. Please record that narrow exception with its scope, risk, follow-up and requested approver in the Governance Exception template, preserving the dependency test isolation.
| input.device = "gpu"; | ||
|
|
||
| #ifdef __GPU_RI | ||
| expect_rejected("device_exx=gpu does not support symmetry=1"); |
There was a problem hiding this comment.
Non-blocking: please retain the effective-symmetry behavior in a repository reader/CLI regression, including omitted symmetry and an MD input that resets symmetry=1 to 0. This test calls check_value directly, so it cannot catch a future change to reset-before-validation ordering. The existing symmetry tests and reported CLI harness provide starting fixtures.
What's changed?
Enable optional GPU acceleration of LCAO Kohn–Sham exact-exchange tensor contractions through LibRI's DDLA CUDA backend.
ENABLE_LIBRI_GPUdefaults toOFF; INPUTdevice_exxdefaults tocpu. To select GPU RI, build withUSE_CUDA=ON,ENABLE_LIBRI_GPU=ON, an explicit compatibleLIBRI_DIRand a CUDA DDLA installation, then set:GPU RI requires LibRI v0.3.0.0 or newer and DDLA's handle-selected batched GEMM API. The verified DDLA revision is
7c0ca08b4349a527cec2a0c0af7d1a1082bd38c7; the original DDLA v0.0.6 release has an incompatible interface. CMake consumesLibDDLAConfig.cmake/ddla::ddlaand checks real/complex GPU API compilation and linkage. GPU RI requires C++17; CPU RI retains its existing dependency selection and C++14 requirement.GPU EXX temporarily rejects effective
symmetry 1, regardless ofexx_symmetry_realspace. The guard reports an error after normal INPUT processing, without changing settings itself; existing resets such as MD setting symmetry to 0 still apply. Usesymmetry 0(time reversal only),symmetry -1(no symmetry), or CPU RI.exx_symmetry_realspaceis an existing parameter; this restriction adds no parameter. Remove the restriction only after corrected dependencies pass application-level symmetry validation.The selector applies to LCAO Kohn–Sham exchange. LR/RDMFT engines retain CPU contractions, including when LR reuses Kohn–Sham exchange geometry. All ranks on a node must see the same allocated GPU list, with at least as many visible GPUs as local ranks; per-rank single-GPU masking is unsupported. Build/install instructions and INPUT help are updated.
Unit Tests and/or Case Tests for my changes
HSE runtime and performance results below are pinned to
f2667c6eed30e661a36bfb0ae1844802c99f171eon SAI16V100. The subsequent integration/test cleanup at6dd58cf762fd730d62eb61ed36e3383e660066bbhas separate targeted validation; the benchmark executable is unchanged.scf_thr=1e-8and report 64 force vectors; forces are enabled, stress and symmetry disabled.Targeted follow-up validation: the original GPU Debug
MODULE_RI_exx_workspacelink failure was reproduced (14 unresolved DDLA symbols); linking its existing target toLibRI::LibRIfixes it. CPU/GPU Debug workspace tests and standalone consumer tests all pass (four CTest invocations, 1/1 each). Real INPUT-reader + EXX-item suites pass 47/47 CPU and 49/49 GPU, with zero skips on a visible V100; masking GPUs skips exactly the two GPU reader cases. Cases cover omitted symmetry, explicit symmetry=1 with real-space reduction disabled, accepted 0/−1, CPU symmetry and MD's normal reset to 0. The existing reader source is moved tounittests/test_read_input_serial.cppwithout changing its tested contents. Runtime OMP/BLAS=1. Combined governance/format checks pass.The duplicate LibRI numerical accumulation test is removed from ABACUS. Upstream LibRI owns its contraction regression; ABACUS retains the small dependency-interface check and HSE application comparisons. No additional INPUT parameter, dependency implementation patch, compatibility shim or custom DDLA finder is added by this cleanup. LR help now reflects the existing CPU workspace behavior. These test/build/help changes do not change HSE runtime algorithms.
Download follow-up validation logs, scripts, patches and source hashes · validation report. Targeted builds and limits are recorded separately from the pinned HSE benchmark.
Raw HSE inputs, outputs and analysis scripts are directly downloadable in the benchmark section below; archive payload hashes and offline reanalysis pass.
Commands recorded in the evidence:
Targeted follow-up commands (full configure flags and return codes are in the validation archive):
The archive includes the per-case INPUT files and expected CLI exit codes. Runtime/INPUT checks use OMP=1; performance measurements explicitly use one MPI rank, OMP=4, BLAS=1 and four bound physical CPU cores per V100.
Earlier release validation at
64b7ef613passed four dependency CTest checks and ten application runs / five backend comparisons covering real/complex tensors, spin polarization, one/two MPI ranks, forces and stress with real-space EXX symmetry disabled. Pinned earlier evidence. Those earlier acceptance results do not override the current broader symmetry restriction. The separate CPU build validated the INPUT target; BSE numerics, HIP, MAGMA, multi-node execution, SOC and long MD are outside this validation.Benchmark results
HSE06 Si64 on SAI
16V100: one V100, one MPI rank/four bound physical CPU cores, OMP=4, BLAS=1; APNS efficiency LCAO, Γ-only, 128 bands, 100 Ry, SCF=1e-8, RI thresholds=1e-4. Fresh atomic starts;symmetry -1, forces enabled, stress disabled. The existingexx_symmetry_realspaceparameter automatically becomes 0 here.Download original inputs and small outputs — 2.57 MiB tar.gz, 490 files · full report · individual timings CSV · numerical comparisons CSV · file/checksum manifest.
The archive contains original INPUT/STRU/KPT files, Si pseudopotential/orbitals, outputs ≤10 MiB/file, scripts and validation logs; larger omissions are listed in the manifest. Downloaded archive SHA256 verified:
4ca994d552094f9e3180751b6572653798d1463ccec9c893c3308f5143fe5587.“beta13 baseline” denotes a baseline built from ABACUS v3.11.0-beta13, with the versions below. Commit IDs distinguish dependency revisions that share a header version string.
4e9f1636b7cab0eff7ad70ab, CPU; header version 2.1.10e8afc393, GPU3c7794c5047d, GPUf2667c6ee, same executablea385ac9), CPU/GPUBefore/after timing composition. Ideal Si64, two starts per variant, paired within two allocations with reversed execution order. Means in seconds; wall time includes initialization, the initial GGA SCF, HSE SCF and complete atomic forces. Three decimals preserve means of timers printed to 0.01 s.
The 9.84× beta13 comparison saves 757.165 s: electronic EXX contributes 44.06%, EXX forces 48.72%, and remaining work 7.22%. It includes dependency/build changes. Baseline → PR CPU LibRI already gives 3.34×, combining ABACUS/dependency/build changes; this is not an isolated LibRI-version comparison. Selecting GPU in that same PR executable then gives 2.95×, with LibXC fixed on CPU and 99.95% of the saving in the two EXX scopes. Baseline → PR GPU LibRI is 9.85× overall.
The paired gains are 10.69×/9.11× for beta13 → experimental and 3.05×/2.85× for PR CPU → GPU; only two repetitions are available. Remainder = wall − electronic EXX − EXX forces. The following nested timers overlap these scopes and must not be added again; effective potential includes LibXC potential.
Calculation consistency. All 10/10 scientific runs finish normally, converge below 1e-8 and output 64 force vectors; 9/9 numerical comparisons pass. Ideal runs retain 12 GGA + 8 HSE iterations. The displaced CPU/GPU pair both use 12 + 12 iterations, with final residual 9.742e-9. The table reports maximum differences across paired runs.
Cross-build limits are 1e-6 eV/atom and 1e-5 eV/Å; same-PR CPU/GPU limits are 1e-8 eV/atom and 1e-6 eV/Å. Force differences are per-atom Euclidean vector norms. The extra pair moves the first Si atom by (+0.01,+0.02,−0.015) Å and has maximum force 0.398013 eV/Å, verifying nontrivial forces; it is not pooled into the ideal-structure timings. Original references and tolerances are unchanged.
Earlier beta13/CP2K and LibRI PR #17 comparisons remain separately documented with their original inputs and outputs.
Known symmetry limitation
Released LibRI v0.3.0.0 gives real/complex Si2 GPU energy and stress differences of 0.0324593 eV and 0.305868 kbar when real-space EXX symmetry is enabled. Its standalone symmetry fixture is outdated, and migrating only the fixture to the Wrapper API still fails GPU contractions.
Source tracing and isolated reproduction identify a later Wrapper regression after the previously merged symmetry fix. A proposed local loop/generator repair passes the standalone real/complex regression; it has not yet been validated in ABACUS. LibRI PR #24 at
d8fc6eca97f6ccf0b4448644f2f891ad7f2eb2e9was separately tested: it reduces failing individual GPU labels from 8/18 to 4/18; real, complex and the combined sum still fail. Removing the four remaining intermediate-atom filter calls in a separate diagnostic source copy passes all 456 checks. This does not establish ABACUS symmetry acceptance. Report, raw sources/logs and diagnostic patch. The guard therefore remains in place. References and tolerances are unchanged.At
6dd58cf762fd730d62eb61ed36e3383e660066bb, all 17 reported CI check runs pass, including the CUDA runtime tests, GNU/Intel builds, CPU tests, pre-commit and governance. The CUDA error 209 failures at earlierf2667c6/853b1644eheads are historical results; the current CUDA run passes. The explicit V100 LibRI validation and its stated symmetry limits remain as documented above.Linked Issue
Reproduction data and timing composition.
Governance Notes
device_exx; both affected INPUT entries document the temporary symmetry restriction. Generated INPUT documentation is not committed.LibRI::LibRIandddla::ddla.unittests/CMakeLists.txt.