First load the following modules
module purgemodule load PrgEnv-gnu/8.3.3module load craype-x86-milanmodule load cray-libsci/23.02.1.1module load cray-fftw/3.3.10.3module load cray-hdf5-parallel/1.12.2.3module load cray-netcdf-hdf5parallel/4.9.0.3Then you have to manually edit the make.inc file and add the " -D__FFTW" flag:
DFLAGS = -D__FFTW -D__MPI -D__SCALAPACK -fallow-argument-mismatchThen you can compile with
make epwFirst load the following modules
module load LUMI/21.12Using the following lumi_gnu.ac file:
FCFLAGS="-ffree-line-length-none -fallow-argument-mismatch -g $FCFLAGS"FFLAGS=" $FFLAGS"Then issue:
export LD_LIBRARY_PATH="$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH"../configure --with-config-file='lumi_gnu.ac' --with-optim-flavor='aggressive'Abinit will then complain that libxc is missing, then go inside the created fallback and compile the fallback. Once this is done, update the lumi_gnu.ac file and re-issue:
./configure --with-config-file='lumi_gnu.ac' --with-optim-flavor='aggressive'Using the following lumi_gnu.ac file:
FCFLAGS="-ffree-line-length-none -fallow-argument-mismatch -g $FCFLAGS"FFLAGS=" $FFLAGS"Then issue:
../configure --with-config-file='lumi_gnu.ac' --with-optim-flavor='aggressive'Abinit will then complain that some libraries are missing, then go inside the created fallback and compile the fallback. Once this is done, update the lumi_gnu.ac file (change the XXX and uncomment) and re-issue:
./configure --with-config-file='lumi_gnu.ac' --with-optim-flavor='aggressive'The issue the following command in your build directory:
../configure FC=mpif90 CC=mpicc --with-netcdf-fortran=/apps/GPP/NETCDF/c-4.9.2_fortran-4.6.1_cxx4-4.3.1_hdf5-1.14.1-2_pnetcdf-1.12.3/GCC/OPENMPI/Note that MareNostrum5 does not let you download things from internet from the cluster. A solution is to download QE 7.4 on your desktop, then compile it on your desktop so it downloads all the necessary dependencies. Then tar it back and copy that on Mn5. You can then untar on the cluster and issue "make clean".
First load the following modules
module purge module load intelmodule load mklmodule load impimodule load fftw/3.3.10First load the following modules
module purgemodule load EasyBuild/4.3.1module load compiler/intel/comp_and_lib/2018.1.163module load intelmpi/5.1.3.181/64module load Python/3.9.6-foss-2018bUsing the following zenobe.ac file:
with_mpi="${I_MPI_ROOT}/intel64"enable_openmp="no"Then issue:
../configure --with-config-file='zenobe.ac' --with-optim-flavor='aggressive'Abinit will then complain some libraries are missing
cd fallbacks./build-abinit-fallbacks.shcd ../../configure --with-config-file='zenobe.ac' --with-optim-flavor='aggressive'You can find configuration script for Zenobe here: https://abinit.github.io/abipy/workflows/manager_examples.html#zenobe
Then to run, one can use:
First load the following modules:
module purgeml load OpenBLAS/0.3.20-GCC-11.3.0Then do:
Then create a "lucia.ac9" file in the build folder with the following lines:
with_libxc=~/ABINIT/abinit-9.8.3/build/fallbacks/install_fb/gnu/11.3/libxc/6.0.0Finally, configure and compile:
../configure --with-config-file='lucia.ac9'Go into the QE folder which should be unpacked somewhere in your $HOME or $PROJECTS_HOME, then execute:
ml PrgEnv-intel./configure --with-scalapack=intel CC=cc FC=ftn MPIF90=ftn FFLAGS="-O2 -assume byterecl -unroll" LDFLAGS="-qmkl" CFLAGS="-O2" SCALAPACK_LIBS="-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" LAPACK_LIBS="-lpthread -lm"make -j 8 epwA job execution – executed somewhere within your $SCRATCH should look like this:
srun --mpi=cray_shasta $PATHQE/bin/pw.x -npool 4 -in scf.in > scf.outFirst load the following modules
module load foss/2019bmodule load HDF5/1.10.5-gompi-2019bmodule load ELPA/2019.11.001-foss-2019bmodule load libxc/4.3.4-GCC-8.3.0Then go inside the QE folder and issue:
./configuremake allCreate a virtual environment at the end of the install. Logout and in again to be in the environment.
Install the code https://phonopy.github.io/phono3py/install.html
conda install -c conda-forge phono3pyLook at the example for Si there https://github.com/phonopy/phono3py/tree/master/example/Si-QE
phono3py --qe -d --dim 2 2 2 -c scf.in --pa autoThis will create all the supercell files. Then create a "header" file for QE that you will aggregate Then run all the calculation, for example using the following submission script
#!/bin/bash##SBATCH --job-name=si-10#SBATCH --output=res.txt#SBATCH --partition=batch##SBATCH --time=01:00:00 # hh:mm:ss#SBATCH --ntasks=36#SBATCH --nodes=1#SBATCH --cpus-per-task=1Then create the force file with phono3py:
phono3py --qe --cf3 Si-{00001..00111}.outphono3py --sym-fcFinally, compute the thermal conductivity and converge the fine grids (i.e. increase the 10 10 10 grid below):
phono3py --mesh 10 10 10 --fc3 --fc2 --brYou should use the local scratch on each node (this implies running on 1 node maximum).
You can use a script similar to this:
#!/bin/bash##SBATCH --job-name=test#SBATCH --output=res.txt#SBATCH --partition=hmem#SBATCH --time=00:20:00#SBATCH --ntasks=6#SBATCH --nodes=1#SBATCH --cpus-per-task=1#####SBATCH --mem-per-cpu=100Using the following discover.ac file:
FC=mpif90CC=mpiccCXX=mpicxxH5CC=h5pccwith_hdf5=/opt/software/hdf/5/1.14.3-intel-openmpiwith_netcdf=/opt/software/netcdf-c/4/4.9.2-intel-openmpiwith_netcdf_fortran=/opt/software/netcdf-fortran/4/4.6.1-intel-openmpiwith_libxc=/opt/software/libxc/5/5.2.2-intel/Then issue:
mkdir buildcd build../configure --with-config-file=discover.ac --with-optim-flavor='aggressive'make -j4Using the following discover.ac file:
FC=mpiifortCC=mpiiccCXX=mpiicpcH5CC=h5ccFCFLAGS=" -I/opt/software/libxc/5/5.2.2-intel/include -I/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/netcdf-c-4.8.1-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include -I/opt/software/szip/2/2.1.1-intel/include -I/opt/software/zlib/1/1.2.11-intel/include -I/opt/software/gnu/gcc-12/isl-0.24/include -I/opt/software/gnu/gcc-12/mpc-1.2.1/include -I/opt/software/gnu/gcc-12/mpfr-4.1.0/include -I/opt/software/gnu/gcc-12/gmp-6.2.1/include -I/opt/software/gnu/gcc-12/gcc-12.1.0/include "FCFLAGS+=" -O2 -xHost "CFLAGS=" -I/opt/software/libxc/5/5.2.2-intel/include -I/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/netcdf-c-4.8.1-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include -I/opt/software/szip/2/2.1.1-intel/include -I/opt/software/zlib/1/1.2.11-intel/include -I/opt/software/gnu/gcc-12/isl-0.24/include -I/opt/software/gnu/gcc-12/mpc-1.2.1/include -I/opt/software/gnu/gcc-12/mpfr-4.1.0/include -I/opt/software/gnu/gcc-12/gmp-6.2.1/include -I/opt/software/gnu/gcc-12/gcc-12.1.0/include "CFLAGS+=" -O2 -xHost "CFLAGS+="-I /opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include/"FCFLAGS+=" -I /opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include/"CPPFLAGS+=" -I /opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include/"with_hdf5=/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/with_netcdf=/opt/software/netcdf-c-4.8.1-intel-hdf5_1.12-api_v112-intel_mpi/with_netcdf_fortran=/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/with_libxc=/opt/software/libxc/5/5.2.2-intel/Then issue:
mkdir buildcd build../configure --with-config-file=discover.ac --with-optim-flavor='aggressive'make -j4First load the following modules:
module purgemodule load python3/latestmodule load intelmodule load mkl/latestmodule load zlib/1/latest-intelmodule load netcdf/c/4.8/4.8.1-intel-hdf5_1.12_api_v18-intel_mpimodule load mpi/latestmodule load gcc/12/latestmodule load gcc/11/latestmodule load netcdf/fortran/4.5/4.5.4-intel-hdf5_1.12_api_v112-intel_mpimodule load tbb/latestmodule load oclfpga/latestmodule load szip/2/latest-intelmodule load compiler-rt/latestmodule load hdf5/1/1.12/1.12.1-intel-zlib_1-szip-api_v112-intel_mpimodule load libxc/5/5.2.2-intelmodule unload --force gcc/11unset CC FC CXX F90Using the following discover.ac file:
FC=mpiifortCC=mpiiccCXX=mpiicpcH5CC=h5ccFCFLAGS=" -I/opt/software/libxc/5/5.2.2-intel/include -I/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/netcdf-c-4.8.1-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include -I/opt/software/szip/2/2.1.1-intel/include -I/opt/software/zlib/1/1.2.11-intel/include -I/opt/software/gnu/gcc-12/isl-0.24/include -I/opt/software/gnu/gcc-12/mpc-1.2.1/include -I/opt/software/gnu/gcc-12/mpfr-4.1.0/include -I/opt/software/gnu/gcc-12/gmp-6.2.1/include -I/opt/software/gnu/gcc-12/gcc-12.1.0/include "FCFLAGS+=" -O2 -xHost "CFLAGS=" -I/opt/software/libxc/5/5.2.2-intel/include -I/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/netcdf-c-4.8.1-intel-hdf5_1.12-api_v112-intel_mpi/include -I/opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include -I/opt/software/szip/2/2.1.1-intel/include -I/opt/software/zlib/1/1.2.11-intel/include -I/opt/software/gnu/gcc-12/isl-0.24/include -I/opt/software/gnu/gcc-12/mpc-1.2.1/include -I/opt/software/gnu/gcc-12/mpfr-4.1.0/include -I/opt/software/gnu/gcc-12/gmp-6.2.1/include -I/opt/software/gnu/gcc-12/gcc-12.1.0/include "CFLAGS+=" -O2 -xHost "CFLAGS+="-I /opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include/"FCFLAGS+=" -I /opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include/"CPPFLAGS+=" -I /opt/software/hdf/5/1.12.1-intel-zlib_1-szip-api_v112-intel_mpi/include/"with_hdf5=/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/with_netcdf=/opt/software/netcdf-c-4.8.1-intel-hdf5_1.12-api_v112-intel_mpi/with_netcdf_fortran=/opt/software/netcdf-fortran-4.5.4-intel-hdf5_1.12-api_v112-intel_mpi/with_libxc=/opt/software/libxc/5/5.2.2-intel/Then issue:
mkdir buildcd build../configure --with-config-file=discover.ac --with-optim-flavor='aggressive'make -j4Go inside qe/:
/configuremake -j 8 epwLoad the following modules:
module purgemodule load cmake/latestmodule load ${COMPILER}module load compiler/latestmodule load blas/latest-${COMPILER}module load lapack/latest-${COMPILER}module load ${MPI}/4/${COMPILER}/latestmodule load fftw/3/latest-${COMPILER}-${MPI}Note: This works but might not be the fastest.
First load the following modules:
module purgemodule load nvidiamodule load nvhpc-nompi/latestmodule load blas/latest-nvidiamodule load lapack/latest-nvidiamodule load openmpi/4/nvidia/latestmodule load fftw/3/latest-nvidia-openmpiGo inside qe/:
./configureThen you need to modify manually the following line in the make.inc file:
DFLAGS = -D__PGI -D__FFTW3 -D__MPI -I/opt/software/fftw/3/3.3.10-nvidia-openmpi/includeSometime it is useful to allow for argument mismatch in make.inc
FFLAGS = -O3 -g -fallow-argument-mismatchYou can use the following compilation argument for debugging: FFLAGS = -g -fallow-argument-mismatch -Wall -funroll-loops -O3 -fgcse-lm -fgcse-sm -ftree-vectorize -fbounds-check -finit-integer=-666 -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-line-length-0First load the following modules
module purgemodule load releases/2020bmodule load imkl/2020.4.304-iimpi-2020b./configure make all