Running EPW on HPC

[smitha20]

Hi all,

I'm currently doing quite a bit of work with EPW on TACC Frontera and was hoping for insight into how the code scales with cores. Currently I'm using ~20 nodes (1120 cores) to do a calculation on Nb and I'm getting very poor CPU usage from the SLURM report and was wondering if maybe I compiled something wrong or am executing the program incorrectly.

Compiled QE + EPW with Intel 19.1.1 (MKL), --with-scalapack=intel, and FFTW3 (3.3.10)
Running 8x8x8 k and 8x8x8 q grids interpolated to 40x40x40 k-fine, 20x20x20 q-fine.

after calculating phonons and using pp.py to grab the necessary files for EPW I run:
*note: ibrun is TACC Frontera's mpirun*
ibrun -np 1120 pw.x -nk 56 -i scf.in > scf.out
ibrun -np 1120 pw.x -nk 1120 -i nscf.in > nscf.out
ibrun -np 1120 epw.x -nk 1120 -i epw.in > epw.out

I'm getting fine results (good spreads on my WFs, not quite converged decays but working on it), but the slurm command seff reveals that I'm only using ~10% CPU usage after the run is complete. Is this normal?

Thanks,

Adam

[smitha20]

Forgot to mention that I'm only doing elph+a2F calculations with auto_projections from SCDM. Please find input files attached.

Adam

######SCF#######
&CONTROL
calculation = 'scf',
etot_conv_thr = 0.0001,
forc_conv_thr = 0.001,
prefix = 'nb',
pseudo_dir = '../pseudo/',
tprnfor = .TRUE.,
tstress = .TRUE.,
verbosity = 'low',
/
&SYSTEM
celldm(1) = 5.6115194567925135,
degauss = 0.035,
ecutrho = 400.0,
ecutwfc = 80.0,
ibrav = 3,
lspinorb = .TRUE.,
nbnd = 30,
noncolin = .TRUE.,
nosym = .FALSE.,
occupations = 'smearing',
smearing = 'gauss',
nat = 1,
ntyp = 1,
/
&ELECTRONS
conv_thr = 1d-12,
diagonalization = 'davidson',
electron_maxstep = 200,
mixing_beta = 0.3,
/
&IONS
/
&CELL
/
ATOMIC_SPECIES
Nb 92.9064 Nb.rel-pbe-spn-kjpaw_psl.1.0.0.UPF
ATOMIC_POSITIONS crystal
Nb 0.000000000000 0.000000000000 0.000000000000
K_POINTS automatic
18 18 18 0 0 0

######NSCF#######
&CONTROL
calculation = 'nscf',
outdir = './',
prefix = 'nb',
pseudo_dir = '../pseudo',
/
&SYSTEM
celldm(1) = 5.6115194567925135,
degauss = 0.035,
ecutrho = 400.0,
ecutwfc = 80.0,
ibrav = 3,
lspinorb = .TRUE.,
nbnd = 30,
noncolin = .TRUE.,
occupations = 'smearing',
smearing = 'gauss',
nat = 1,
ntyp = 1,
/
&ELECTRONS
conv_thr = 1d-12,
diagonalization = 'davidson',
electron_maxstep = 200,
mixing_beta = 0.3,
/
&IONS
/
&CELL
/
ATOMIC_SPECIES
Nb 92.9064 Nb.rel-pbe-spn-kjpaw_psl.1.0.0.UPF
ATOMIC_POSITIONS crystal
Nb 0.000000000000 0.000000000000 0.000000000000
K_POINTS crystal
512
0.00000000 0.00000000 0.00000000 1.9531e-03
0.00000000 0.00000000 0.12500000 1.9531e-03
0.00000000 0.00000000 0.25000000 1.9531e-03
0.00000000 0.00000000 0.37500000 1.9531e-03
0.00000000 0.00000000 0.50000000 1.9531e-03
....

######EPW#######
--
&inputepw
prefix = 'nb',
outdir = './'

elph = .true.
epbwrite = .false.
epbread = .false.
etf_mem = 1

epwwrite = .true.
epwread = .false.

nbndsub = 12
bands_skipped = 'exclude_bands = 1-8'

wannierize = .true.
num_iter = 500
dis_froz_min= 19.46
dis_froz_max= 33.89

restart = .true.

auto_projections = .true.
scdm_proj = .true.
scdm_mu = 26.86
scdm_sigma = 9.011921196574042
scdm_entanglement = 'erfc'

wdata(1) = 'bands_plot = .true.'
wdata(2) = 'begin kpoint_path'
wdata(3) = 'G 0.00 0.00 0.00 H 0.50 -0.50 0.00'
wdata(4) = 'H 0.50 -0.50 0.00 N 0.00 0.00 0.50'
wdata(5) = 'N 0.00 0.00 0.50 G 0.00 0.00 0.00'
wdata(6) = 'G 0.00 0.00 0.00 P 0.25 0.25 0.25'
wdata(7) = 'P 0.25 0.25 0.25 H 0.50 -0.50 0.50'
wdata(8) = 'end kpoint_path'
wdata(9) = 'bands_plot_format = gnuplot xmgrace'
wdata(10) = 'precond = .true.'
wdata(11) = 'use_ws_distance = .true.'
wdata(12) = 'dis_num_iter = 200'
iverbosity = 2

elecselfen = .true.
phonselfen = .true.
! delta_approx = .true.


fsthick = 7.0 ! eV
degaussw = 0.005 ! eV
degaussq = 0.05 !meV
temps = 0.05 ! K

a2f = .true.

dvscf_dir = '../phonons/save'

nk1 = 8
nk2 = 8
nk3 = 8

nq1 = 8
nq2 = 8
nq3 = 8

nkf1 = 40
nkf2 = 40
nkf3 = 40

nqf1 = 20
nqf2 = 20
nqf3 = 20
/