I am suspicious of the large size of the complex-valued array of epmatwp_mem defined in the subroutine of epmatwp_mem.
In you case, it is 16 * 24 * 24 * 16391 * (24*3) = 10876280832 bytes = about 11 GB.
However, you first have to check the line 1366 of bloch2wan.f90:
Thank you for looking into this! Line 1366 of bloch2wan.f90 simply says “len1( = zero” which doesn’t appear helpful (at least to me). It seems like that subroutine in bloch2wan.f90 was called by line 514 of ephwann_shuffle.f90 which says:
IF (etf_mem > 0) THEN
CALL ephbloch2wanp_mem(nbndsub, nmodes, xqc, nqc, irvec_k, irvec_g, nrr_k, nrr_g)
ENDIF
I have etf_mem = 1 which should have more I/O and less memory requirement. I might try to set etf_mem = 0 to avoid this call, but it seems counterintuitive since that would require more memory. What do you think?
Line 1366 of bloch2wan.f90 simply says “len1( = zero” which doesn’t appear helpful (at least to me).
If the line 1366 of bloch2wan.f90 is the statement of “len1( = zero”, the first statement which is related to the writing to memory of the subroutine of epmatwp_mem, my suspicion is right.
The solution to this issue is related to the details of cluster and environment you are using; you had better consult with the system administrator.
It is also related to your current memory usage. So the simple solution with “ulimit -s unlimited” might not work.
I have etf_mem = 1 which should have more I/O and less memory requirement. I might try to set etf_mem = 0 to avoid this call, but it seems counterintuitive since that would require more memory. What do you think?
I think that etf_mem=0 (etf_mem=2 as well) doesn’t work.
Instead you could try the allocatable array for epmatwp_mem in the subroutine of epmatwp_mem.
If it also doesn’t work, you might need more modifications in the code.
Thank you very much for your advice! It’s unfortunate that I might have to consult my system administrator, which is usually a very slow process. I’d like to try a couple of things before submitting a help ticket.
Do you think simply increasing the number of nodes might solve this problem? I don’t seem to have this problem for a smaller calculation (12-atom cell vs. 24-atom cell) so do you think it would be worth the try?
You say
Instead you could try the allocatable array for epmatwp_mem in the subroutine of epmatwp_mem.
Could you explain a bit more how I would do this? How would I need to modify the code (bloch2wan.f90)?
Do you think simply increasing the number of nodes might solve this problem? I don’t seem to have this problem for a smaller calculation (12-atom cell vs. 24-atom cell) so do you think it would be worth the try?
It might work, but also might not work; I think that in your case there is no solution which works with 100% probability since it is related to several things such as system details, build environments, and current memory usage, etc.
You say
Instead you could try the allocatable array for epmatwp_mem in the subroutine of epmatwp_mem.
Could you explain a bit more how I would do this? How would I need to modify the code (bloch2wan.f90)?
You can try the allocatable array for epmatwp_mem with the correct statement of (de)allocate as below:
Thank you very much! I will try these things as soon as I am able to run calculations again on Frontera, which unfortunately is reserved for a week for large-scale jobs (1000s of nodes). I will let you know what happens. Thank you!
= zero” which doesn’t appear helpful (at least to me). It seems like that subroutine in bloch2wan.f90 was called by line 514 of ephwann_shuffle.f90 which says: