run in parallel with wannierize = .true. , WF don't converge

[dacheng]

Dear Samuel and Carla,
I am using the EPW code. When the wannierize is set as true, it will first run the wannier90 code. To my knowledge, the wannier90 code is only in seriel, so if you run EPW in seriel, this would be OK, but if you run EPW in parallel (with wannierize = .true.), there might have some unpredicted results. In fact, I have tested this, and found that the WF centre and spread do not converge (when run EPW in parellel with wannierize = .true.). I also run EPW in seriel with wannierize = .true. (using the same epw.in file) and it truns out that the WF centre and spread converges well. Is it that there are some special setting in using EPW in parallel with wannierize = .true. [when running in parallel, I have used the the same command line as that in the tests of EPW.tar.gz]

with many thanks!
Dacheng

[roxana]

Dear Dacheng,

You need to make sure the same number of cores and pools are used when run the nscf and epw calculations.

You may want to take a look at the following topic:
viewtopic.php?f=6&t=146&start=10

Best,
Roxana

[dacheng]

[quote="roxana"]
Dear roxana,
Very thanks for your reply.
Yes I'm using the same cores and pools when run nscf and epw calculations (both cores and pools are set as 24).
when run in parallel, below is the WF centre and spread (for 24152 cycles)
------------------------------------------------------------------------------
Cycle: 24152
WF centre and spread 1 ( 0.000122, -2.860142, -0.000097 ) 1.20853996
WF centre and spread 2 ( 0.000001, -0.000252, 0.000363 ) 1.60046364
WF centre and spread 3 ( -0.825694, 1.430107, 5.960567 ) 1.14444243
WF centre and spread 4 ( -0.082424, 0.034572, 8.877901 ) 7.54579700
WF centre and spread 5 ( -0.085151, 0.050339, 8.887579 ) 7.79906977
WF centre and spread 6 ( -0.085829, 0.040474, 8.881744 ) 7.83191193
WF centre and spread 7 ( -0.066779, 0.033734, 8.888278 ) 7.67781423
WF centre and spread 8 ( -0.044410, 0.015743, 8.782482 ) 8.09832981
WF centre and spread 9 ( -0.000163, 0.000511, 0.000772 ) 1.14946031
WF centre and spread 10 ( 0.009308, -0.022357, 1.930597 ) 1.35257466
WF centre and spread 11 ( -0.017857, -0.004132, 1.945960 ) 1.36118202
WF centre and spread 12 ( 0.002004, 0.021340, 2.017698 ) 1.23378023
WF centre and spread 13 ( 0.018964, 0.002217, 15.922170 ) 2.03662335
WF centre and spread 14 ( -0.320415, -0.556480, 11.510364 ) 12.86903918
WF centre and spread 15 ( 0.030173, 0.004836, 15.857244 ) 1.93150642
WF centre and spread 16 ( 0.010709, -0.018697, 15.924339 ) 1.99236594
Sum of centres and spreads ( -1.457442, -1.828187,115.387959 ) 66.83290089


while when run in seriel, for 3000 cycles, the WF centre and spread is
------------------------------------------------------------------------------
Cycle: 3000
WF centre and spread 1 ( 2.476879, -1.430027, 0.000001 ) 1.26675035
WF centre and spread 2 ( 0.000003, 0.000008, 0.000001 ) 1.46496706
WF centre and spread 3 ( -0.825627, 1.430028, 5.960148 ) 1.12843950
WF centre and spread 4 ( -0.000000, 0.000003, 8.940222 ) 1.95175155
WF centre and spread 5 ( -0.000000, 0.000001, 8.940222 ) 1.96293025
WF centre and spread 6 ( 0.000000, 0.000000, 8.940222 ) 1.96112394
WF centre and spread 7 ( -0.000000, 0.000002, 8.940222 ) 2.02812399
WF centre and spread 8 ( 0.000001, -0.000004, 8.940222 ) 2.03000523
WF centre and spread 9 ( -0.000948, 0.018638, 2.041492 ) 0.99400148
WF centre and spread 10 ( 0.001812, -0.018767, 2.041474 ) 0.99403029
WF centre and spread 11 ( -0.000459, 0.000070, 1.844384 ) 1.33326642
WF centre and spread 12 ( -0.000000, 0.000001, -0.000000 ) 1.12127871
WF centre and spread 13 ( 0.000474, 0.000096, 16.036055 ) 1.33327299
WF centre and spread 14 ( -1.651254, 0.000004, 11.920294 ) 1.30965998
WF centre and spread 15 ( 0.001145, 0.018612, 15.838956 ) 0.99400444
WF centre and spread 16 ( -0.002024, -0.018777, 15.838970 ) 0.99405282
Sum of centres and spreads ( 0.000001, -0.000113,116.222883 ) 22.86765899

you can see that the centres do not differ much, while the spreads are in big difference.

Best,

dacheng

[sponce]

Dear dacheng,

A few people on the Forum have reported similar issue (try to find their post on the forum).
In all cases so far, it turns out it was a usage mistake and not a bug in EPW.

I can suggest you the following:
1) You do not need that many cycle. Everything should be convergence in less than 1000 cycles. If not, it will not converge. You can therefore set the max cycle to 1000.
2) be sure to use the latest QE+EPW version (version 4.1) as there was indeed a bug in version 4.0.
3) be sure to use:
mpirun -np X path/pw.x -npool X < nscf.in
mpirun -np X path/epw.x -npool X < epw.in
4) create two empty directory to do the test sequential versus parallel to be sure that you are not using the wrong WF in seq. that has been generate in parallel.
So 1 directory for seq and 1 for parallel.

If it still does not work, you could consider posting your nscf and epw input files here.

Cheers,

Samuel

[dacheng]

[quote="sponce"]

Dear Samuel,
Thanks for your reply.
1) I have change the num_iter = 1000
2) I am using the qe-6.0 with EPW-4.1
3) when run in parallel, the command line is
mpirun -np 24 path/pw.x -npool 24 < nscf.in
mpirun -np 24 path/epw.x -npool 24 < epw.in
4) I have create a new directory to run it in parallel, while the WF still do not converge
Final State
WF centre and spread 1 ( -0.652736, -0.538270, -0.038223 ) 7.25297359
WF centre and spread 2 ( -0.005876, -0.003262, -0.009514 ) 2.77697510
WF centre and spread 3 ( -0.522591, -0.192233, 5.990744 ) 8.28278460
WF centre and spread 4 ( -0.067034, 0.004677, 8.801800 ) 7.13416346
WF centre and spread 5 ( -0.076167, 0.009628, 8.687760 ) 9.65813363
WF centre and spread 6 ( -0.088199, 0.012509, 8.590836 ) 11.46057800
WF centre and spread 7 ( -0.039752, 0.011602, 8.640811 ) 12.89728793
WF centre and spread 8 ( -0.069445, 0.017570, 8.791373 ) 8.45199204
WF centre and spread 9 ( -0.049657, -0.097572, 1.656488 ) 6.14607874
WF centre and spread 10 ( -0.038225, -0.198892, 1.642233 ) 5.12946655
WF centre and spread 11 ( -0.083093, -0.113423, 1.690049 ) 5.20818278
WF centre and spread 12 ( -0.116637, -0.168390, 1.645858 ) 6.07314421
WF centre and spread 13 ( 0.047916, 0.014597, 16.030401 ) 7.22585125
WF centre and spread 14 ( -0.016689, -0.034435, 15.991783 ) 9.62696018
WF centre and spread 15 ( -0.010410, -0.011012, 16.200572 ) 5.96130828
WF centre and spread 16 ( 0.015658, -0.027670, 16.172520 ) 5.09212924
Sum of centres and spreads ( -1.772936, -1.314575,120.485491 ) 118.37800959

Below is my nscf.in and epw.in file

&control
calculation='nscf'
pseudo_dir = '../pp/',
outdir='./'
prefix='PdCoO2'
wf_collect = .false.
/
&system
nosym = .true.
ibrav = 0,
celldm(1)=1.8897,
nat=4,
ntyp=3,
ecutwfc = 60,
ecutrho = 600,
starting_magnetization = 0.0,
occupations='smearing',
smearing='mp',
degauss=0.01
/
&electrons
diagonalization='cg'
diago_david_ndim = 8
conv_thr = 1.0d-10
mixing_beta = 0.7
/
CELL_PARAMETERS
1.651276595 0.000000000 5.960230489
-0.825638297 1.430047480 5.960230489
-0.825638297 -1.430047480 5.960230489
ATOMIC_SPECIES
Pd 106.42 Pd.pbe-n-rrkjus_psl.0.2.2.UPF
Co 58.9332 Co.pbe-nd-rrkjus.UPF
O 15.9994 O.pbe-rrkjus.UPF
ATOMIC_POSITIONS (crystal)
Pd -0.000000000 -0.000000000 0.000000000
Co 0.500000000 0.500000000 0.500000000
O 0.112975898 0.112975898 0.112975898
O 0.887024102 0.887024102 0.887024102
K_POINTS crystal
216
0.00000000 0.00000000 0.00000000 4.629630e-03
0.00000000 0.00000000 0.16666667 4.629630e-03
0.00000000 0.00000000 0.33333333 4.629630e-03
...........................................................................
...........................................................................
0.83333333 0.83333333 0.50000000 4.629630e-03
0.83333333 0.83333333 0.66666667 4.629630e-03
0.83333333 0.83333333 0.83333333 4.629630e-03

epw.in file
&inputepw
prefix = 'PdCoO2'
amass(1) = 106.42
amass(2) = 58.9332
amass(3) = 15.9994
outdir = './'

elph = .true.
kmaps = .false.
epbwrite = .true.
epbread = .false.

epwwrite = .true.
epwread = .false.

nbndsub = 16
nbndskip = 20

wannierize = .true.
num_iter = 1000
dis_win_max = 21
dis_win_min = -9
dis_froz_min= 0
dis_froz_max= 13
proj(1) = 'Pd:l=0;l=2,mr=1'
proj(2) = 'Co:l=0;l=2'
proj(3) = 'O:sp3'

wdata(1) = 'bands_plot = .true.'
wdata(2) = 'begin kpoint_path'
wdata(3) = 'L 0.00 0.50 0.00 Z 0.50 0.50 0.50'
wdata(4) = 'Z 0.50 0.50 0.50 G 0.00 0.00 0.00'
wdata(5) = 'G 0.00 0.00 0.00 F 0.50 0.50 0.00'
wdata(6) = 'end kpoint_path'
wdata(7) = 'bands_plot_format = gnuplot'

iverbosity = 0

elecselfen = .true.
phonselfen = .true.
a2f = .true.

parallel_k = .true.
parallel_q = .false.
fsthick = 6 ! eV
eptemp = 300 ! K
degaussw = 1.0 ! eV
degaussq = 0.05 ! meV

dvscf_dir = '../ph_t/save'
filukk = './PdCoO2.ukk'

nkf1 = 12
nkf2 = 12
nkf3 = 12

nqf1 = 12
nqf2 = 12
nqf3 = 12

nk1 = 6
nk2 = 6
nk3 = 6

nq1 = 3
nq2 = 3
nq3 = 3
/
6 cartesian
0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 1.0
-0.672880070304275E-01 -0.116546246888716E+00 0.186420829426988E-01 1.0
-0.134576014060855E+00 0.000000000000000E+00 0.372841658853977E-01 1.0
0.138777878078145E-16 0.233092493777433E+00 -0.346944695195361E-17 1.0
0.000000000000000E+00 0.000000000000000E+00 0.559262488168070E-01 1.0
0.134576014060855E+00 0.233092493777433E+00 0.186420829314093E-01 1.0

Thank you again!

Best

dacheng

[roxana]

Dear Dacheng,

I noticed that the pseudopotentials you are using are US and EPW has only been tested for NC. This could be the source of the problem.

Best,
Roxana