Skip to content

feat(lcao): add optional LibRI GPU contractions through DDLA - #8169

Open
Stardust0831 wants to merge 29 commits into
deepmodeling:developfrom
Stardust0831:experiment/lcao-hse-gpu
Open

Stardust0831 wants to merge 29 commits into
deepmodeling:developfrom
Stardust0831:experiment/lcao-hse-gpu

Conversation

@Stardust0831

@Stardust0831 Stardust0831 commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

What's changed?

Enable optional GPU acceleration of LCAO Kohn–Sham exact-exchange tensor contractions through LibRI's DDLA CUDA backend. ENABLE_LIBRI_GPU defaults to OFF; INPUT device_exx defaults to cpu. To select GPU RI, build with USE_CUDA=ON, ENABLE_LIBRI_GPU=ON, an explicit compatible LIBRI_DIR and a CUDA DDLA installation, then set:

device      gpu
device_exx  gpu
symmetry    -1

GPU RI requires LibRI v0.3.0.0 or newer and DDLA's handle-selected batched GEMM API. The verified DDLA revision is 7c0ca08b4349a527cec2a0c0af7d1a1082bd38c7; the original DDLA v0.0.6 release has an incompatible interface. CMake consumes LibDDLAConfig.cmake / ddla::ddla and checks real/complex GPU API compilation and linkage. GPU RI requires C++17; CPU RI retains its existing dependency selection and C++14 requirement.

GPU EXX temporarily rejects effective symmetry 1, regardless of exx_symmetry_realspace. The guard reports an error after normal INPUT processing, without changing settings itself; existing resets such as MD setting symmetry to 0 still apply. Use symmetry 0 (time reversal only), symmetry -1 (no symmetry), or CPU RI. exx_symmetry_realspace is an existing parameter; this restriction adds no parameter. Remove the restriction only after corrected dependencies pass application-level symmetry validation.

The selector applies to LCAO Kohn–Sham exchange. LR/RDMFT engines retain CPU contractions, including when LR reuses Kohn–Sham exchange geometry. All ranks on a node must see the same allocated GPU list, with at least as many visible GPUs as local ranks; per-rank single-GPU masking is unsupported. Build/install instructions and INPUT help are updated.

Unit Tests and/or Case Tests for my changes

HSE runtime and performance results below are pinned to f2667c6eed30e661a36bfb0ae1844802c99f171e on SAI 16V100. The subsequent integration/test cleanup at 6dd58cf762fd730d62eb61ed36e3383e660066bb has separate targeted validation; the benchmark executable is unchanged.

  • At the benchmark source: complete GPU executable builds, CPU/GPU serial EXX INPUT suites pass 33/33 tests each, and 8/8 CLI checks cover GPU rejection with symmetry 1 and true/false/default real-space settings, implicit defaults, CPU symmetry, and GPU symmetry 0/−1. Governance and changed-file formatting checks pass.
  • Fresh Si64 benchmark: 10/10 scientific runs and 9/9 energy/force comparisons pass, including a displaced structure with maximum force 0.398013 eV/Å. Same-executable CPU/GPU differences are at most 9.664e-13 eV/atom and 3.163e-10 eV/Å across these cases. The displaced pair has force-vector difference 1.415e-10 eV/Å. All calculations converge below scf_thr=1e-8 and report 64 force vectors; forces are enabled, stress and symmetry disabled.
  • The source tree, executable, inputs, pseudopotential/orbital files and runtime placement were audited. A driver assertion and a mistakenly duplicated recovery attempt are retained separately; the completed ten-run sample and all failed-attempt records are preserved.

Targeted follow-up validation: the original GPU Debug MODULE_RI_exx_workspace link failure was reproduced (14 unresolved DDLA symbols); linking its existing target to LibRI::LibRI fixes it. CPU/GPU Debug workspace tests and standalone consumer tests all pass (four CTest invocations, 1/1 each). Real INPUT-reader + EXX-item suites pass 47/47 CPU and 49/49 GPU, with zero skips on a visible V100; masking GPUs skips exactly the two GPU reader cases. Cases cover omitted symmetry, explicit symmetry=1 with real-space reduction disabled, accepted 0/−1, CPU symmetry and MD's normal reset to 0. The existing reader source is moved to unittests/test_read_input_serial.cpp without changing its tested contents. Runtime OMP/BLAS=1. Combined governance/format checks pass.

The duplicate LibRI numerical accumulation test is removed from ABACUS. Upstream LibRI owns its contraction regression; ABACUS retains the small dependency-interface check and HSE application comparisons. No additional INPUT parameter, dependency implementation patch, compatibility shim or custom DDLA finder is added by this cleanup. LR help now reflects the existing CPU workspace behavior. These test/build/help changes do not change HSE runtime algorithms.

Download follow-up validation logs, scripts, patches and source hashes · validation report. Targeted builds and limits are recorded separately from the pinned HSE benchmark.

Raw HSE inputs, outputs and analysis scripts are directly downloadable in the benchmark section below; archive payload hashes and offline reanalysis pass.

Commands recorded in the evidence:

cmake --build build/abacus-gpu --target abacus_std_gpu MODULE_IO_read_item_serial --parallel 28
cmake --build build/abacus-cpu --target MODULE_IO_read_item_serial --parallel 24
ctest --test-dir build/abacus-gpu -V -R '^MODULE_IO_read_item_serial$'
ctest --test-dir build/abacus-cpu -V -R '^MODULE_IO_read_item_serial$'
abacus_std_gpu --version
abacus_std_gpu -h device_exx
abacus_std_gpu --check-input
python3 analyze.py
python3 audit-assets.py
python3 verify-analysis.py

Targeted follow-up commands (full configure flags and return codes are in the validation archive):

cmake --build build/abacus-gpu-debug --target MODULE_RI_exx_workspace --parallel 24 --verbose
ctest --test-dir build/abacus-gpu-debug -V -R '^MODULE_RI_exx_workspace$'
ctest --test-dir build/abacus-cpu-debug -V -R '^MODULE_RI_exx_workspace$'
ctest --test-dir build/consumer-cpu -V
ctest --test-dir build/consumer-gpu -V
cmake --build build/final-gpu --target MODULE_IO_read_input_serial MODULE_IO_read_item_serial --parallel 32
ctest --test-dir build/final-cpu -V -R '^MODULE_IO_read_(input|item)_serial$'
ctest --test-dir build/final-gpu -V -R '^MODULE_IO_read_(input|item)_serial$'

The archive includes the per-case INPUT files and expected CLI exit codes. Runtime/INPUT checks use OMP=1; performance measurements explicitly use one MPI rank, OMP=4, BLAS=1 and four bound physical CPU cores per V100.

Earlier release validation at 64b7ef613 passed four dependency CTest checks and ten application runs / five backend comparisons covering real/complex tensors, spin polarization, one/two MPI ranks, forces and stress with real-space EXX symmetry disabled. Pinned earlier evidence. Those earlier acceptance results do not override the current broader symmetry restriction. The separate CPU build validated the INPUT target; BSE numerics, HIP, MAGMA, multi-node execution, SOC and long MD are outside this validation.

Benchmark results

HSE06 Si64 on SAI 16V100: one V100, one MPI rank/four bound physical CPU cores, OMP=4, BLAS=1; APNS efficiency LCAO, Γ-only, 128 bands, 100 Ry, SCF=1e-8, RI thresholds=1e-4. Fresh atomic starts; symmetry -1, forces enabled, stress disabled. The existing exx_symmetry_realspace parameter automatically becomes 0 here.

Download original inputs and small outputs — 2.57 MiB tar.gz, 490 files · full report · individual timings CSV · numerical comparisons CSV · file/checksum manifest.

The archive contains original INPUT/STRU/KPT files, Si pseudopotential/orbitals, outputs ≤10 MiB/file, scripts and validation logs; larger omissions are listed in the manifest. Downloaded archive SHA256 verified: 4ca994d552094f9e3180751b6572653798d1463ccec9c893c3308f5143fe5587.

“beta13 baseline” denotes a baseline built from ABACUS v3.11.0-beta13, with the versions below. Commit IDs distinguish dependency revisions that share a header version string.

Build ABACUS LibRI LibXC
beta13 baseline 4e9f1636b7ca b0eff7ad70ab, CPU; header version 2.1.1 7.0.0, CPU
beta13 + GPU libraries same beta13 revision + recorded adapter patch 0e8afc393, GPU 3c7794c5047d, GPU
PR, CPU/GPU LibRI f2667c6ee, same executable v0.3.0.0 (a385ac9), CPU/GPU 7.0.0, CPU

Before/after timing composition. Ideal Si64, two starts per variant, paired within two allocations with reversed execution order. Means in seconds; wall time includes initialization, the initial GGA SCF, HSE SCF and complete atomic forces. Three decimals preserve means of timers printed to 0.01 s.

Component / s beta13 baseline beta13 + GPU libraries PR, CPU LibRI PR, GPU LibRI
Electronic EXX 359.895 26.295 104.265 24.980
EXX forces 398.015 29.105 115.645 28.020
Remaining wall time 84.920 30.265 32.680 32.600
Complete calculation 842.830 85.665 252.590 85.600
Comparison Overall speedup Electronic EXX saved / s EXX forces saved / s Remainder saved / s
beta13 baseline → beta13 + GPU libraries 9.84× 333.600 368.910 54.655
beta13 baseline → PR CPU LibRI 3.34× 255.630 282.370 52.240
PR CPU → GPU LibRI 2.95× 79.285 87.625 0.080

The 9.84× beta13 comparison saves 757.165 s: electronic EXX contributes 44.06%, EXX forces 48.72%, and remaining work 7.22%. It includes dependency/build changes. Baseline → PR CPU LibRI already gives 3.34×, combining ABACUS/dependency/build changes; this is not an isolated LibRI-version comparison. Selecting GPU in that same PR executable then gives 2.95×, with LibXC fixed on CPU and 99.95% of the saving in the two EXX scopes. Baseline → PR GPU LibRI is 9.85× overall.

The paired gains are 10.69×/9.11× for beta13 → experimental and 3.05×/2.85× for PR CPU → GPU; only two repetitions are available. Remainder = wall − electronic EXX − EXX forces. The following nested timers overlap these scopes and must not be added again; effective potential includes LibXC potential.

Nested timer / s beta13 baseline beta13 + GPU libraries PR, CPU LibRI PR, GPU LibRI
Effective potential 30.355 10.160 13.695 13.700
LibXC potential 20.250 4.025 8.165 8.185
LCAO solver 31.090 7.990 6.555 6.590

Calculation consistency. All 10/10 scientific runs finish normally, converge below 1e-8 and output 64 force vectors; 9/9 numerical comparisons pass. Ideal runs retain 12 GGA + 8 HSE iterations. The displaced CPU/GPU pair both use 12 + 12 iterations, with final residual 9.742e-9. The table reports maximum differences across paired runs.

Comparison max ΔE / eV max ΔE / eV/atom max atom-wise ‖ΔF‖ / eV/Å Result
beta13 baseline → GPU libraries 1.388e-07 2.169e-09 1.273e-09 PASS
beta13 baseline → PR CPU LibRI 1.455e-11 2.274e-13 2.236e-10 PASS
beta13 baseline → PR GPU LibRI 4.729e-11 7.390e-13 4.583e-10 PASS
PR CPU → GPU, ideal Si64 6.185e-11 9.663e-13 3.162e-10 PASS
PR CPU → GPU, displaced Si64 3.638e-12 5.684e-14 1.414e-10 PASS

Cross-build limits are 1e-6 eV/atom and 1e-5 eV/Å; same-PR CPU/GPU limits are 1e-8 eV/atom and 1e-6 eV/Å. Force differences are per-atom Euclidean vector norms. The extra pair moves the first Si atom by (+0.01,+0.02,−0.015) Å and has maximum force 0.398013 eV/Å, verifying nontrivial forces; it is not pooled into the ideal-structure timings. Original references and tolerances are unchanged.

Earlier beta13/CP2K and LibRI PR #17 comparisons remain separately documented with their original inputs and outputs.

Known symmetry limitation

Released LibRI v0.3.0.0 gives real/complex Si2 GPU energy and stress differences of 0.0324593 eV and 0.305868 kbar when real-space EXX symmetry is enabled. Its standalone symmetry fixture is outdated, and migrating only the fixture to the Wrapper API still fails GPU contractions.

Source tracing and isolated reproduction identify a later Wrapper regression after the previously merged symmetry fix. A proposed local loop/generator repair passes the standalone real/complex regression; it has not yet been validated in ABACUS. LibRI PR #24 at d8fc6eca97f6ccf0b4448644f2f891ad7f2eb2e9 was separately tested: it reduces failing individual GPU labels from 8/18 to 4/18; real, complex and the combined sum still fail. Removing the four remaining intermediate-atom filter calls in a separate diagnostic source copy passes all 456 checks. This does not establish ABACUS symmetry acceptance. Report, raw sources/logs and diagnostic patch. The guard therefore remains in place. References and tolerances are unchanged.

At 6dd58cf762fd730d62eb61ed36e3383e660066bb, all 17 reported CI check runs pass, including the CUDA runtime tests, GNU/Intel builds, CPU tests, pre-commit and governance. The CUDA error 209 failures at earlier f2667c6/853b1644e heads are historical results; the current CUDA run passes. The explicit V100 LibRI validation and its stated symmetry limits remain as documented above.

Linked Issue

Reproduction data and timing composition.

Governance Notes

  • INPUT metadata, help, validation and MPI broadcast cover device_exx; both affected INPUT entries document the temporary symmetry restriction. Generated INPUT documentation is not committed.
  • INPUT tests use the existing serial target and public registry accessor; no new test access macros are introduced.
  • Core impact: select the backend before LCAO EXX initialization and encapsulate dependencies in LibRI::LibRI and ddla::ddla.
  • The removed standalone numerical test carried direct MPI initialization/finalization calls; these are absent from the final PR, so no exception for them is needed. Reader tests reuse the serial target, with the migrated source registered in unittests/CMakeLists.txt.
  • Follow-up: add DDLA installation to the existing toolchain workflow.

@userchba

Copy link
Copy Markdown

the LibDDLA version which is compatible with LibRI GPU is https://github.com/userchba/LibDDLA/releases/tag/v0.0.4, current LibRI GPU can't correctly compute symmetry situation which LibRI CPU can compute correctly

@mohanchen mohanchen added the GPU & DCU & HPC GPU and DCU and HPC related any issues label Oct 18, 2026

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of f07d8d14f39d26782490cc4bc15d0a58c5700e62 with independent source review and adjudication. No build/runtime/MPI/GPU tests were run; CI diagnostics were unavailable.

Comment thread docs/advanced/install.md Outdated
(default OFF), `USE_CUDA=ON`, and a CUDA build of DDLA. Like other dependent
options, GPU RI is disabled when its LibRI/CUDA prerequisites are disabled.
INPUT `exx_device` defaults to `cpu`, retaining CPU contractions even with
`device gpu`. To use GPU contractions, set both `device gpu` and `exx_device gpu`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] State the GPU visibility requirement for MPI runs

For two ranks on one node, exposing one different GPU per rank through CUDA_VISIBLE_DEVICES passes ABACUS device setup but makes the pinned LibRI validate_local_gpu_count() throw at the first GPU contraction (local_size=2 > device_count=1). This backend therefore requires each process to see at least as many GPUs as there are node-local ranks, a constraint missing here. Please document it with a compatible launch layout, or extend the compatibility patch to respect the existing per-rank binding.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guide and device_exx help now require a common visible GPU list with at least one GPU per node-local rank, and the guide includes a compatible two-rank launch. This addresses the documentation request; per-rank single-GPU masking remains outside the supported layout.

Comment thread source/source_lcao/setup_exx.cpp Outdated
Exx_Info& exx_info)
{
#ifdef __EXX
const bool use_gpu = inp.exx_device == "gpu";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Please clarify the selector's scope for independent LR/RDMFT exchange engines. Standalone LR, fresh LR-kernel setup after KS, and RDMFT construct their own Exx_LRI objects, which retain CPU mode despite accepted exx_device=gpu; matching-kernel KS pointer reuse retains GPU mode. If those engines are in scope, pass the selection to them; otherwise document or reject the unsupported combinations. A follow-up after merge is fine for these additional workflows.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: current share_exx_lri() creates a fresh CPU workspace and shares geometry only, so the GPU-inheritance sentence in device_exx help, the installation guide and the PR body is stale. Please state that LR remains on CPU even when it reuses KS geometry.

{
return 1;
}
lri.cal_mode = RI::LRI_Cal_Mode::GPU;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Consider an opt-in numerical regression for the patch's signed accumulation in a follow-up after merge. This consumer switches cal_mode but then executes only CPU nrm2, and the configure probe only compiles/links. Alongside the reported force/stress runs, a tiny real/complex CPU-versus-GPU case with nonempty output and repeated +1/-1 contractions would preserve the specific accumulation fix across dependency updates.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new opt-in libri_gpu_accumulation test checks all 18 contractions for real and complex tensors using a nonzero copied seed and repeated -1/+1 accumulation against the CPU result. This addresses the requested committed regression coverage. It is enabled with ENABLE_LIBRI_GPU_TESTS=ON.

@Stardust0831
Stardust0831 marked this pull request as ready for review October 19, 2026 14:30
@mohanchen mohanchen added the Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS label Oct 20, 2026
@mohanchen

Copy link
Copy Markdown
Collaborator

To be consistent, I think device_exx maybe better than exx_device

@mohanchen mohanchen added Input&Output Suitable for coders without knowing too many DFT details EXX/lr-TDDFT/RPA/BSE/GW Related to beyond DFT methods labels Oct 20, 2026
@Stardust0831

Copy link
Copy Markdown
Collaborator Author

To be consistent, I think device_exx maybe better than exx_device

Thank you for the review and suggestions; I’ve renamed exx_device to device_exx.

@QuantumMisaka

QuantumMisaka commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

@PeizeLin @maki49 May you have a look? Thanks !

@mohanchen

Copy link
Copy Markdown
Collaborator

@Growl1234 Could you check the cmake changes for this PR?

Comment thread cmake/modules/FindLibRI.cmake Outdated
if(ENABLE_LIBRI_GPU)
message(
FATAL_ERROR
"ENABLE_LIBRI_GPU requires an explicitly supplied, patched LIBRI_DIR; "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that LibRI master branch hasn't yet accepted the LibRI changes? If this is true then this PR should not be merged on abacus side first.

Comment thread cmake/modules/FindDDLA.cmake Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LibDDLA has CMake config export, why is FindDDLA module needed here?

@maki49

maki49 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

I have no further comment within this PR's scope.
LR and symmetry with LibRI on GPU are already supported in my local branch but they need to update LibRI. I will make a new PR in the future when everything is ready.

@Stardust0831

Copy link
Copy Markdown
Collaborator Author

An update on the LibRI dependency fixes:

The ABACUS test results reported in this PR still use the temporary compatibility patch. The new direct-accumulation implementation has passed LibRI-level GPU regression tests with DDLA v0.0.4, but this PR has not yet been updated to use it or retested with full ABACUS HSE calculations.

I plan to move this PR back to draft while the remaining LibRI fix is pending upstream. Once merged, I will update the dependency, remove the compatibility patch, address the remaining review comments, and sync with develop. I will then rerun the HSE single-point energy, force, and stress regressions before marking this PR ready for review again.

@Stardust0831
Stardust0831 marked this pull request as ready for review November 21, 2026 04:39

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GPU-enabled main test target MODULE_RI_exx_workspace needs its DDLA link dependency; the inline comments also cover two nonblocking follow-ups. This review is source-only: no build or test was run, author-reported results remain attributed, and CI was not independently checked.

Comment thread source/CMakeLists.txt
foreach(
_abacus_feature_target IN
ITEMS nlohmann_json::nlohmann_json
LibRI::LibRI

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Carry the DDLA link dependency into the RI workspace test

With BUILD_TESTING=ON, ENABLE_LIBRI_GPU=ON and CMAKE_BUILD_TYPE=Debug, these copied requirements enable __GPU_RI in MODULE_RI_exx_workspace, whose link list has no LibRI::LibRI or DDLA dependency. Its cal_Hs()/cal_force() calls instantiate runtime GPU dispatch even when the objects default to CPU, leaving DDLA references unresolved in an unoptimized link. Please link this target to LibRI::LibRI, or supply a consistent CPU-only compile interface, and verify that specific main-project target.

{
// This standalone dependency test has no ABACUS runtime to initialize MPI.
int provided = 0;
if (MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided) != MPI_SUCCESS)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the standalone-test rationale explains these direct MPI_Init_thread/MPI_Finalize calls. Please record that narrow exception with its scope, risk, follow-up and requested approver in the Governance Exception template, preserving the dependency test isolation.

input.device = "gpu";

#ifdef __GPU_RI
expect_rejected("device_exx=gpu does not support symmetry=1");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: please retain the effective-symmetry behavior in a repository reader/CLI regression, including omitted symmetry and an MD input that resets symmetry=1 to 0. This test calls check_value directly, so it cannot catch a future change to reset-before-validation ordering. The existing symmetry tests and reported CLI harness provide starting fixtures.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS EXX/lr-TDDFT/RPA/BSE/GW Related to beyond DFT methods GPU & DCU & HPC GPU and DCU and HPC related any issues Input&Output Suitable for coders without knowing too many DFT details

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants