Memory consumption Wannier representation of the EP-matrix
Posted: Tue Jun 07, 2016 2:13 pm
After having used EPW for some time now, I am working on more complicated systems and I get into memory issues.
Currently I have a somewhat (apparently) large system: 18 atoms and 178 non-pseudo electrons. It is a 2d system with a kpoint grid of 18x18x1 kpoints. The wannierisation works well after some trial and error. However, when the ep-matrix elements are converted to wannier representation, EPW quickly starts to use a lot of memory (more than the 64GB that I have available on my nodes). I do not understand where this memory consumption comes from, I have 185 bands, 106 wannier functions, 18x18x1 kpoints, 18x3=54 modes and only one q point at Gamma. Most likely the consumption comes from epmatwe ( nbndsub, nbndsub, nrr_k, nmodes) and epmatwp ( nbndsub, nbndsub, nrr_k, nmodes), although they should only be ~200M elements in size.
Am I missing something conceptually in my assessment of the memory requirements?
A solution for me would be to run the program for a specific mode at a time, which seems to be also a scientifically relevant use of EPW (viewtopic.php?f=3&t=20). I tried to do this by performing the phonon calculation only for a specific mode and although EPW handles this correctly, it still defines the matrices for nmodes = 3*nat.
Is there any other solution that you could think of? We are pretty close to publishing some nice results, apart from not being able to check the convergence with respect to the coarse grid kpoint size.
Edit:
It seems there is a typo in the wigner_seitz.f90 and wigner_sietz2.f90 files.
The maximum number of elements is set to 20*nk1*nk3*nk3 instead of 20*nk1*nk2*nk3. Since I was trying this with a kpoint-grid of 36x36x1 (a Cu111 1x1 supercell), the array got out of bounds(although check bounds somehow did not catch this error directly, hence the long time it took me to get back to you). This seems to be inherited from the previous version of epw, as epw-3 also contains this error.
The performance increase of epw/QE in this new version is quite satisfying.
Currently I have a somewhat (apparently) large system: 18 atoms and 178 non-pseudo electrons. It is a 2d system with a kpoint grid of 18x18x1 kpoints. The wannierisation works well after some trial and error. However, when the ep-matrix elements are converted to wannier representation, EPW quickly starts to use a lot of memory (more than the 64GB that I have available on my nodes). I do not understand where this memory consumption comes from, I have 185 bands, 106 wannier functions, 18x18x1 kpoints, 18x3=54 modes and only one q point at Gamma. Most likely the consumption comes from epmatwe ( nbndsub, nbndsub, nrr_k, nmodes) and epmatwp ( nbndsub, nbndsub, nrr_k, nmodes), although they should only be ~200M elements in size.
Am I missing something conceptually in my assessment of the memory requirements?
A solution for me would be to run the program for a specific mode at a time, which seems to be also a scientifically relevant use of EPW (viewtopic.php?f=3&t=20). I tried to do this by performing the phonon calculation only for a specific mode and although EPW handles this correctly, it still defines the matrices for nmodes = 3*nat.
Is there any other solution that you could think of? We are pretty close to publishing some nice results, apart from not being able to check the convergence with respect to the coarse grid kpoint size.
Edit:
It seems there is a typo in the wigner_seitz.f90 and wigner_sietz2.f90 files.
The maximum number of elements is set to 20*nk1*nk3*nk3 instead of 20*nk1*nk2*nk3. Since I was trying this with a kpoint-grid of 36x36x1 (a Cu111 1x1 supercell), the array got out of bounds(although check bounds somehow did not catch this error directly, hence the long time it took me to get back to you). This seems to be inherited from the previous version of epw, as epw-3 also contains this error.
The performance increase of epw/QE in this new version is quite satisfying.