3.3. LLVM¶
This section describes how to use the C/C++/Fortran compilers based on LLVM. LLVM, which generates binaries for compute nodes, is available on both the login nodes (Intel) and the compute nodes.
3.3.1. Usage of LLVM compilers¶
3.3.1.1. Environment setting¶
These compiler is located at:
On login node (cross environment) /vol0004/apps/r/OSS_CN/llvm-21.1.0/cross_clangfx On compute node (native environment) /vol0004/apps/r/OSS_CN/llvm-21.1.0/own_clangfx
(If you prefer to use the previous version, ‘lvm-v19.1.4’ remains available.)
Execute the following command to setup the environment before running compilers.
login node
[_LNIlogin]$ module load LLVM/llvmorg-21.1.0
compute node
[_CNlogin]$ module load LLVM/llvmorg-21.1.0
Available languages are following
Software
Language
Version
Command
Clang
C
21.1.0
clang
Clang++
C++
21.1.0
clang++
Flang
Fortran
21.1.0
flang
3.3.1.2. How to Use¶
Here is an example of how to compile C. The commands are common for both nodes. For compilation options, please refer to those provided by LLVM’s Clang/Flang.
In particular, with flang, the compile process may sometimes take longer. In such cases, please allocate a larger cache size on the compute node, as described below.
[Sequential execution]
- login node
[_LNIlogin]$ clang -O3 source_file
- compute node
[_CNlogin]$ clang -O3 source_file
[OpenMP]
Please add “-fopenmp” option for OpenMP library.
- login node
[_LNIlogin]$ clang -O3 -fopenmp source_file
- compute node
[_CNlogin]$ clang -O3 -fopenmp source_file
[MPI]
You can use MPI library via mpiclang/mpiclang++/mpiflang commands.
- login node
[_LNIlogin]$ mpiclang -O3 -fopenmp source_file
- compute node
[_CNlogin]$ mpiclang -O3 -fopenmp source_file
See also
The same procedure applies when using C++ (mpiclang++) and Fortran (mpiflang).
Example: Compilation in a job script
#!/bin/sh -x #PJM -L "node=1" #PJM -L "rscgrp=small" #PJM -L "elapse=01:00:00" #PJM -x PJM_LLIO_GFSCACHE=/vol0004 #PJM -g groupname #PJM --llio cn-cache-size=1Gi #PJM -s # module purge module load LLVM/llvmorg-21.1.0 mpiclang -o sample1.out sample1.c
3.3.1.3. How to execute¶
The compiled binary can be executed either from an interactive job or from a script. Here is an example of script. To improve performance, transfer the executable and LLVM libraries using llio_transfer.
#!/bin/sh -x #PJM -L "node=128" #PJM --mpi "max-proc-per-node=4" #PJM -L "rscgrp=small" #PJM -L "elapse=01:00:00" #PJM -x PJM_LLIO_GFSCACHE=/vol0004 #PJM -g groupname #PJM -s # module purge module load LLVM/llvmorg-21.1.0 llio_transfer `find ${LLVM_BASEDIR}/lib64 -type f -name \*.so\*` llio_transfer `find ${MPI_HOME}/lib64 -type f -name \*.so\*` llio_transfer sample1.out mpiexec ./sample1.out
See also
For C++ and Fortran programs, the same execution procedure should be followed.
3.3.1.4. Using Libraries¶
With this LLVM, the OpenBLAS and FFTW libraries are available for use.
Please note that this library cannot be used with other compilers.
3.3.1.4.1. Usage of OpenBLAS¶
- The libraries are located at the following path:
Library
Path of Library
LP64, sequential
/vol0004/apps/r/OSS_CN/llvm/openblas-seq
LP64, thread-parallel
/vol0004/apps/r/OSS_CN/llvm/openblas-omp
ILP64, sequential
/vol0004/apps/r/OSS_CN/llvm/openblas-seq-ilp64
ILP64, thread-parallel
/vol0004/apps/r/OSS_CN/llvm/openblas-omp-ilp64
3.3.1.4.1.1. How to compile¶
When compiling a program that uses OpenBLAS, please first configure the environment for LLVM. Then, add the option -lopenblas at the linking stage.
For multi-threaded execution, please also add the option -fopenmp.
In addition, please add the following options depending on the programming language.
For C/C++:
Please specify the library and header files of the OpenBLAS installation directory (${OPENBLAS_DIR}) using the -L and -I options.
Please add -lflang_rt.runtime.
For Fortran:
Please specify the library files of the OpenBLAS installation directory (${OPENBLAS_DIR}) using the -L option.
3.3.1.4.1.2. How to execute¶
When running a program that uses OpenBLAS, please add ${OPENBLAS_DIR}/lib to the LD_LIBRARY_PATH.
Modifications from the original OpenBLAS
Compared with the original OpenBLAS 0.3.26, the following routines have been tuned for A64FX.
DGEMM
SGEMM
3.3.1.4.2. Usage of FFTW¶
FFTW libraries are located at /vol0004/apps/r/OSS_CN/llvm/fftw3. (FFTW_DIR)
When compiling a program that uses FFTW, please first configure the environment for LLVM. Then, add the following options at the linking stage:
Single-precision sequential library : -lfftw3f -lm
Single-precision multi-threaded library : -lfftw3f_omp -lfftw3f -lm
Double-precision sequential library : -lfftw3 -lm
Double-precision multi-threaded library : -lfftw3_omp -lfftw3 -lm
When using the multi-threaded library, please add the option -fopenmp.
In addition, please add the following options depending on the programming language:
For C/C++:
Please specify the library and header files of the FFTW installation directory (${FFTW_DIR}) using the -L and -I options.
Please add -lflang_rt.runtime.
For Fortran:
Please specify the library files of the FFTW installation directory (${FFTW_DIR}) using the -L option.
3.3.1.4.2.1. How to execute¶
When running a program that uses FFTW, please add ${FFTW_DIR}/lib to the LD_LIBRARY_PATH.
Modifications from the original FFTW
Header files supporting various SIMD instructions have been adapted for SVE instructions.
In the automatic tuning mechanism called plan generation, efficiency is achieved by removing processing routes that are not selected.
3.3.1.5. Cautions¶
3.3.1.5.1. Caution for REAL(KIND=16) and COMPLEX(KIND=16) types¶
On the cross-compilation environment flang available on the login nodes, the following error occurs when compiling Fortran programs that use REAL(KIND=16) or COMPLEX(KIND=16) types.
Example for the failure
flang complex16.f90 error: Semantic errors in complex16.f90 ./complex16.f90:3:3: error: COMPLEX(KIND=16) is not an enabled type for this target complex(kind=16) :: x ^^^^^^^^^^^^^^^^^^^^^flang real16.f90 error: Semantic errors in real16.f90 ./real16.f90:3:3: error: REAL(KIND=16) is not an enabled type for this target real(kind=16) :: x, y, z ^^^^^^^^^^^^^^^^^^^^^^^^
Workaround
When using these types, please compile in the native environment (i.e., on the compute nodes).
3.3.1.5.2. Notes for Interlanguage Linkage¶
3.3.1.5.2.1. Compilation Command and Required Options during Linking¶
When linking combined Fortran, C, and C++ object programs created with the LLVM compiler, please note the following points.
- When linking C++ objects (Using libstdc++)
When linking using the command clang or flang, please specify the option
-lstdc++.- When linking Fortran objects
When linking using the command clang or clang++, please specify the following options.
L${LLVM_BASEDIR}/lib64 -lflang_rt.runtime
3.3.2. Migration from Fujitsu compiler¶
This chapter explains the differences in specifications and behavior between the Fujitsu compiler and the LLVM (https://llvm.org/) compiler environment to facilitate the smooth migration of applications developed and built with the Fujitsu compiler to the LLVM environment.
3.3.2.1. Compiler Commands¶
Here describes about differences between the Fujitsu compiler and the LLVM compiler. For the Fujitsu compiler, the command names differ between native compilers and cross compilers. In contrast, the LLVM compiler uses the same command names regardless of whether they are native or cross compilers.
Item |
Fujitsu compiler |
LLVM compiler |
|---|---|---|
C compiler |
fcc
fccpx
|
clang |
C++ compiler |
FCC
FCCpx
|
clang++ |
Fortran compiler |
frt
frtpx
|
flang |
3.3.2.2. Compiler Options¶
Fujitsu compilers (trad mode, clang mode) and LLVM compilers differ in their compile option systems and how to specify their options. This section presents a comparison table of compiler options for both compilers as reference information for migration.
Table: Compiler Options with an Identical Function lists compiler options that share the same name and have identical functional descriptions in both the Fujitsu compiler and the LLVM compiler. Table: Compiler Options with a Similar Function (For C/C++Compiler) and Table: Compiler Options with a Similar Function (For Fortran Compiler) list compiler options that have different names but identical functional descriptions.
Note
Fujitsu compiler-specific options(Example:-K or -ffj) are not available in the LLVM compiler. Specifying these will result in a compilation error and cause the process to terminate. For examples of Fujitsu compiler-specific options, please refer to Incompatible features.
Even if options have the same functional overview, optimization results may differ. After migration, we recommend performing functional verification.
Option Name |
|---|
-Dname[=tokens] |
-E |
-Idir |
-Ldir |
-P |
-S |
-Uname |
-Wtool,arg1[,arg2]… |
-c |
-g |
-g0 |
-lname |
-o pathname |
-shared |
-v |
-w |
Fujitsu Compiler trad mode Option Names |
Fujitsu Compiler clang mode Option Names |
LLVM 21 clang/clang++ Option Names |
|---|---|---|
-KA64FX |
-mcpu=a64fx |
-mcpu=a64fx |
-KARMV8_1_A |
-march=armv8.1-a |
-march=armv8.1-a |
-KARMV8_2_A |
-march=armv8.2-a |
-march=armv8.2-a |
-KARMV8_3_A |
-march=armv8.3-a |
-march=armv8.3-a |
-KARMV8_A |
-march=armv8-a |
-march=armv8-a |
-KGENERIC_CPU |
-mcpu=generic |
-mcpu=generic |
-K{PIC|pic} |
-f{PIC|pic} |
-f{PIC|pic} |
-Kcmodel={small|large} |
-mcmodel={small|large} |
-mcmodel={small|large} |
-K{eval|noeval} |
-f{fast-math|no-fast-math} |
-f{fast-math|no-fast-math} |
-Kfast |
-Ofast |
-O3 -ffast-math |
-Kfp_contract |
-ffp-contract=fast |
-ffp-contract={fast|on|fast-honor-pragmas} |
-Knofp_contract |
-ffp-contract=off |
-ffp-contract=off |
-Kfp_relaxed |
-ffj-fp-relaxed |
-freciprocal-math or -mrecip |
-Knofp_relaxed |
-ffj-no-fp-relaxed |
-fno-reciprocal-math or -mrecip=none |
-K{largepage|nolargepage} |
-ffj-{largepage|no-largepage} |
Refer to Large Page Function |
-K{lib|nolib} |
-f{builtin|no-builtin} |
-f{builtin|no-builtin} |
-K{omitfp|noomitfp} |
-f{omit-frame-pointer|no-omit-frame-pointer} |
-f{omit-frame-pointer|no-omit-frame-pointer} |
-K{openmp|noopenmp} |
-f{openmp|no-openmp} |
-f{openmp|no-openmp} |
-K{openmp_simd|noopenmp_simd} |
-f{openmp-simd|no-openmp-simd} |
-f{openmp-simd|no-openmp-simd} |
-Koptmsg=2 |
-Rpass=.* |
-Rpass=.* |
-K{simd|nosimd} |
-f{vectorize|no-vectorize} |
-f{vectorize|no-vectorize} |
-K{strict_aliasing|nostrict_aliasing} |
-f{strict-aliasing|no-strict-aliasing} |
-f{strict-aliasing|no-strict-aliasing} |
-Kswp |
-ffj-swp |
-mllvm -aarch64-enable-pipeliner |
-K{unroll|nounroll} |
-f{unroll-loops|no-unroll-loops} |
-f{unroll-loops|no-unroll-loops} |
-N{exceptions|noexceptions} |
-f{exceptions|no-exceptions} |
-f{exceptions|no-exceptions} |
-V |
--version |
--version |
{-x-|-x0} |
-f{inline-functions|no-inline-functions} |
-f{inline-functions|no-inline-functions} |
Fujitsu Compiler trad mode Option Names |
LLVM 21 clang/clang++ Option Names |
|---|---|
-KA64FX |
-mcpu=a64fx |
-KARMV8_1_A |
-march=armv8.1-a |
-KARMV8_2_A |
-march=armv8.2-a |
-KARMV8_3_A |
-march=armv8.3-a |
-KARMV8_A |
-march=armv8-a |
-KGENERIC_CPU |
-mcpu=generic |
-K{PIC|pic} |
-f{PIC|pic} |
-Kcmodel={small|large} |
-mcmodel={small|large} |
-K{eval|noeval} |
-f{fast-math|no-fast-math} |
-Kfast |
-O3 -ffast-math -fstack-arrays |
-Kfp_contract |
-ffp-contract={fast|on} |
-Knofp_contract |
-ffp-contract=off |
-Kfp_relaxed |
-freciprocal-math or -mrecip |
-Knofp_relaxed |
-fno-reciprocal-math or -mrecip=none |
-K{largepage|nolargepage} |
Refer to Large Page Function |
-K{omitfp|noomitfp} |
-f{omit-frame-pointer|no-omit-frame-pointer} |
-K{openmp|noopenmp} |
-f{openmp|no-openmp} |
-Koptmsg=2 |
-Rpass=.* |
-K{simd|nosimd} |
-f{vectorize|no-vectorize} |
-Kswp |
-mllvm -aarch64-enable-pipeliner |
-K{unroll|nounroll} |
-f{unroll-loops|no-unroll-loops} |
-V |
--version |
3.3.2.3. Incompatible features¶
The LLVM compiler does not support Fujitsu compiler and A64FX-specific features. Main incompatible features are listed below.
Fujitsu Compiler-Specific Options
Specifying Fujitsu compiler-specific compiler options such as the following to the LLVM compiler will result in a compilation error and cause the process to terminate.
-K option (Example: -Kfast)
-N option (Example: -NRtrap)
-ffj options (Example: -ffj-fjprof)
Options related to the math libraries(Example: -SSL2、-lfjlapacksve)
Other Fujitsu compiler-specific options（Example: -A’name(token)’）
For options with similar functionality, refer to Table: Compiler Options with a Similar Function (For C/C++Compiler) and Table: Compiler Options with a Similar Function (For Fortran Compiler) in Compiler Options.
Fujitsu Compiler-Specific Optimization Control Specifiers
If Fujitsu compiler-specific optimization controlspecifiers are written within the source code, they will not be recognized by the LLVM compiler and will be ignored.
Fujitsu Compiler-Specific Environment Variables
The following Fujitsu compiler-specific environment variables that used during compilation with the Fujitsu compiler and when executing binaries created by the Fujitsu compiler has no effect on operation with the LLVM compiler and they are ignored.
Environment variable for compile command(Example: fcc_ENV)
Environment variable for execution(Example: FLIB_C_MESSAGE)
Predefined Macro Names
Fujitsu compilers predefine macros such as
__FUJITSUor__CLANG_FUJITSU, which may be used in conditional branching within source code. These macros are not defined in LLVM compilers. If you are using them for conditional branching, we recommend verifying that the expected processing occurs.A64FX-Specific Features (Available through environment variables or options)
The following features were primarily controlled in Fujitsu compilers through environment variables, compiler options, and optimization control specifiers; however, these control mechanisms are not provided in LLVM compilers. Applications that utilized these features may experience operational differences during migration; therefore, we recommend verifying their functionality through operational testing.
Hardware barrier (Example: Environment variables FLIB_BARRIER=HARD)
Sector cache function (Example: Optimization control specifier scache_isolate_way L2=n1)
Hardware prefetch assistance function (Example: -Khpctag)
3.3.2.4. Substitute¶
3.3.2.4.1. Large Page Function¶
The Fujitsu compiler had large page functionality enabled by default, making it available without requiring any special configuration. This may result in improved performance depending on the environment and application. On the other hand, this feature is not enabled by default in the LLVM compiler, so explicit specification is required to use it.
In the Fugaku environment, you can use the large page feature of the HPC extension by adding the following to the compiler (linker) options.
-L/opt/FJSVxos/mmm/lib64 -lmpg -Wl,-T/opt/FJSVxos/mmm/util/bss-2mb.lds
Specify this option before any other libraries. For details on the Large Page function, refer to the manual “Job Operation Software End-user’s Guide for HPC Extensions”.
Compilation Example:
clang a.c -L/opt/FJSVxos/mmm/lib64 -lmpg \
Wl,-T/opt/FJSVxos/mmm/util/bss-2mb.lds
Note
The memory allocation behavior in the LLVM compiler may differ from that in the Fujitsu compiler. We recommend verifying that the migration functions as intended through actual builds and operational testing.
3.3.2.5. Object and Source Incompatibility¶
3.3.2.5.1. Object Incompatible¶
Object files, assembler source files, module files (.mod) and etc. generated by Fujitsu compilers (trad mode or clang mode) and LLVM compilers are incompatible. Therefore, all source code must be recompiled with the LLVM compiler.
Although Fujitsu clang mode is LLVM-based, it includes Fujitsu’s unique extensions, and its behavior differs from pure LLVM. Even if partial compatibility appears to exist, reusing objects is not recommended as it may lead to unintended behavior.
3.3.2.5.2. Source Incompatibility¶
The LLVM compiler adheres to standard language specifications (C/C++/Fortran), but proprietary intrinsic functions and service routines that were available with Fujitsu compilers are not supported. Source code that depends on these may require modifications when using LLVM.
3.3.2.5.3. Execution Result Incompatibility¶
Even with identical source code, there may be differences in execution results between Fujitsu compilers and LLVM compilers. This incompatibility primarily stems from the following two points. During post-migration verification, it is recommended to confirm numerical consistency and output format compatibility.
- Differences in numerical calculation precision and errorDifferences in compiler optimization and implementations of numerical calculation libraries may lead to variations in the precision and error of numerical calculations.
- Differences in output specificationsThe appearance and format of output results, such as floating-point display format (number of digits, exponential notation, handling of spaces, etc.), may differ.
3.3.2.6. About Filename Suffixes¶
Indicates whether filename suffixes used with Fujitsu compilers (trad mode/clang mode) are available for use with LLVM compilers (clang, clang++, flang).
C compiler clang can compile files with the suffixes listed in the table below as input.
File type |
File suffix |
|---|---|
C source file |
.c |
C source file |
.i |
Assembler source file |
.s |
Assembler source file that must be preprocessed |
.S |
Object file |
.o |
C++ compiler clang++ can compile files with the suffixes listed in the table below as input.
File type |
File suffix |
|---|---|
C++ source file |
.cc |
C++ source file |
.CC |
C++ source file |
.cpp |
C++ source file |
.CPP |
C++ source file |
.cxx |
C++ source file |
.CXX |
C++ source file |
.c++ |
C++ source file |
.C++ |
C++ source file |
.C |
C++ source file |
.cp |
C++ source file |
.ii |
Assembler source file |
.s |
Assembler source file that must be preprocessed |
.S |
Object file |
.o |
Fortran compiler flang can compile files with the suffixes listed in the table below as input. The file format (fixed format/free format) is automatically determined based on the file extension, so no special option specification is required.
Assembler source files (with the .s extension) cannot be compiled with flang and will cause errors during compilation. Please use clang or clang++ for compiling.
File type |
File suffix |
Format |
|---|---|---|
Fortran source program |
.f |
Fixed Format |
Fortran source program |
.for |
Fixed Format |
Fortran source program |
.f90 |
Free Format |
Fortran source program |
.f95 |
Free Format |
Fortran source program |
.f03 |
Free Format |
Fortran source program |
.f08 |
Free Format |
Preprocessing directive Fortran source program |
.F |
Fixed Format |
Preprocessing directive Fortran source program |
.FOR |
Fixed Format |
Preprocessing directive Fortran source program |
.F90 |
Free Format |
Preprocessing directive Fortran source program |
.F95 |
Free Format |
Preprocessing directive Fortran source program |
.F03 |
Free Format |
Preprocessing directive Fortran source program |
.F08 |
Free Format |
Object file |
.o |
- |
3.3.2.7. SIMD Built-in Functions¶
Fujitsu Compiler (clang mode) supports the SIMD built-in functions documented in “Arm C Language Extensions” version 00bet1 (First public release).
The LLVM compiler (clang) is progressively implementing support for SIMD built-in functions based on the ACLE (Arm C Language Extensions) specification published by Arm. However, it does not fully comply with the latest ACLE specification, and due to specification changes or implementation differences, it may not function as expected. Therefore, when migrating, we recommend verifying whether the SIMD built-in functions you intend to use are supported in the LLVM environment through actual builds and functional verification.
For details on SIMD built-in functions, please refer to the documentation published by Arm for developers. The specifications can be found on GitHub at <https://github.com/ARM-software/acle/releases>.
3.3.2.8. Compatibility with GNU Specifications (C/C++ and Fortran)¶
The LLVM compilers (clang and flang) support some of the extensions and compatibility options available in the GNU C/C++ and GNU Fortran compilers. Many features used in Fujitsu compilers (trad/clang mode) are likely available in the LLVM environment as well. However, specification changes or implementation differences may affect behavior. Therefore, during migration, we recommend verifying that the extended syntax and compatibility options you use function as intended in the LLVM environment through actual builds and functional testing.
3.3.2.9. Language Specifications¶
3.3.2.9.1. Supported Language Standard¶
The following table shows the differences in the language standard supported by the Fujitsu compiler and the LLVM compiler. The notation “(partial)” indicates that the compiler in question does not support all features of the specified standard, but only provides partial compliance.
For Fujitsu compilers, the scope of support is defined as part of the product’s operational guarantee and technical support coverage, For LLVM compilers, the support status is indicated based on the presence or absence of implemented features.
For details on the status of each feature, please refer to the following documentation page. (Based on information as of September 2025)
- Fujitsu Compiler Support Status“Supercomputer Fugaku User Guide - Language and development environment”
- LLVM Compiler (clang/flang) Support StatusC language<https://clang.llvm.org/c_status.html>C++ language<https://clang.llvm.org/cxx_status.html>
Item |
Fujitsu Compiler trad mode |
Fujitsu Compiler clang mode |
LLVM Compiler |
|---|---|---|---|
C Language Standard |
C11 Standard
C99 Standard
C89 Standard
|
C11 Standard
C99 Standard
C89 Standard
|
C23 Standard (part)
C17 Standard (part)
C11 Standard (part)
C99 Standard
C89 Standard
|
C++ Language Standard |
C++17 Standard (part)
C++14 Standard
C++11 Standard
C++03 Standard
|
C++17 Standard
C++14 Standard
C++11 Standard
C++03 Standard
|
C++23 Standard (part)
C++20 Standard (part)
C++17 Standard
C++14 Standard
C++11 Standard
C++03 Standard
|
Fortran Language Standard |
Fortran2018 Standard (part)
Fortran2008 Standard
Fortran2003 Standard
Fortran95 Standard
Fortran90 Standard
Fortran77 Standard
|
None |
Fortran2023 Standard (part)
Fortran2018 Standard (part)
Fortran2008 Standard (part)
Fortran2003 Standard (part)
Fortran95 Standard
Fortran90 Standard
Fortran77 Standard
|
3.3.2.9.2. OpenMP API Specifications¶
The following table shows the differences in the supported OpenMP API specification. The notation “(partial)” indicates that the compiler in question does not support all features of the specified standard, but only provides partial compliance. For specifications without “(Part)”, only the latest supported version based on official documentation is listed.
For Fujitsu compilers, the scope of support is defined as part of the product’s operational guarantee and technical support. For LLVM compilers, the support status is indicated based on the presence or absence of implemented features.
Item |
Fujitsu Compiler trad mode |
Fujitsu Compiler clang mode |
LLVM Compiler |
|---|---|---|---|
C/C++ OpenMP |
OpenMP 4.5 (part)
OpenMP 4.0 (part)
OpenMP 3.1
|
OpenMP 5.0 (part)
OpenMP 4.5
|
OpenMP 5.1 (part)
OpenMP 5.0 (part)
OpenMP 4.5
|
Fortran OpenMP |
OpenMP 5.0 (part)
OpenMP 4.5
|
なし |
OpenMP 4.0 (part)
OpenMP 3.1
|
For the support status of clang/flang, please refer to the page below. The table reflects information as of September 2025.
- OpenMP support of clang
- OpenMP support of flang