Bridge++

Home > Bridge++

Bridge++ is available at https://bridge.kek.jp/Lattice-code/ .

You may want to link the following packages (included in the tar ball of Bridge++), of which building is also stated in this page.

  • lime (need to read ILDG format configurations)
  • QWS (a solver for Clover fermion; if your local lattice size is 32 x YY x ZZ x TT, the multigrid solver becomes much faster.

If you are not sure, preparing lime is a good idea.

Building on Fugaku

  1. Download the tar ball (assumes bridge-2.x.y.tar.gz) and expand it:
    tar -zxf bridge-2.x.y.tar.gz cd bridge-2.x.y
  2. prepare the lime (optional but needed to read ILDG format):
    cd extra
    tar -xzf lime-1.3.2.tar.gz cd lime-1.3.2
    ./configure CC=fccpx CFLAGS=-Nclang --host=armv8
    make
    cd ../../
  3. prepare the QWS (optional):
    cd extra tar -zxf qws_20210322_0f95_bridge.tar.gz
    cd qws-modified
    make
    cd ../../
  4. edit Makefile as follow:
    • 1-a. Target paltform: comment out “target = PC_GNU” and uncomment “target = Fugaku_CLANG”
    • 1-b. Communicaiton: set “use_mpi = fjmpi”
    • 1-c. Multi-threading: set “use_thread = omp”
    • 1-e. Compiler optimization level: set “use_opt_level = high”
    • 1-j. Alternative code: set “use_alternative = yes”
      • set “use_alt_qxs = yes”
      • set “use_qxs_arch = acle”
    • 3-a. LIME: if you want to use lime, set “use_lime_library = yes”
    • 3-g. QWS: if you want to use QWS, set “use_qws_library = yes”
  5. build:
    make

Running the tests

test of the core part of the library (“corelib”)

  • The binary is build/tests/bridge.elf.
  • The script would be something like (assumes 1 MPI process):
    #!/bin/bash #PJM -L "node=1"
    #PJM -L "rscgrp=small"
    #PJM -L "elapse=1:00:00"
    #PJM --mpi "proc=1"
    #PJM -g YOUR_GROPU

    export FLIB_BARRIER=HARD # optional but improves the performance in multi thread case
    mpiexec ./bridge.elf --all

test of Fugaku native solvers

  • build the tests
    cd test_alt_spectrum/src/
    make
  • run the test. The binary is test_alt_spectrum/test/bridge.elf
  • a job script would be
    #!/bin/bash
    #PJM -L "node=1"
    #PJM -L "rscgrp=small"
    #PJM -L "elapse=1:00:00"
    #PJM --mpi "proc=4"
    #PJM -g YOUR_GROPU

    export OMP_NUM_THREADS=12
    export FLIB_BARRIER=HARD # optional but improves the performance in multi thread case
    mpiexec ./bridge.elf

Restrictions

Fugaku native code (alt_QXS) has the following restriction on the local lattice volume. The local volume (the volume in each MPI proceess) must be a multiple of the following minimal volume.

  • double precision case: 4x2x1x1
  • single precision case: 4x4x1x1
  • double precision case with even-odd operators: 8x2x1x1 (x-direction is doubled)
  • single precision case with even-odd operators: 8x4x1x1 (x-direction is doubled)
  • to use QWS: 32x1x1x1

he above minimal shape (except for QWS) can be changed in x-y directions can be modified by editing the following parameters in src/lib_alt_QXS/inline_ACLE/define_vlen.h
#define VLENXD 4
#define VLENYD 2

#define VLENXS 4
#define VLENYS 4

The name of these parameters is self-explaining. Note that VLENXD x VLENYD must be 8 and VLENXS x VLENYS must be 16.

Tips

  • If you want to use lime installed in the system (or your own build in the other directory), specify the suitable path in 3-a. LIME section of Makefile.
  • For debugging, set the debug flag at 1-g. : use_debug = yes
  • make lib skipps buildings the test of corelib part.
  • For a large scale of job, the rankmap tool at https://github.com/RIKEN-LQCD/rankmap4d may help to improve the performance. Bridge++ uses the lexical ordering so use “rankmap_4d_general” or “rankmap_4d”.