I am using the structural plasticity module to simulate large-scale
networks on HPC. The way I run these simulations with MPI creates a
large number of small data files (.npy) which I organize into a HDF5
file in a post hoc manner. This has hitherto not been a problem,
however, I recently increased the size of my simulations and the number
of files that I create exceeds my HPCinode quota. So, I have to come up
with a way of handling all these files or the way I export them.
Ideally, I would like to concurrently write to a file (like zarr) with
multiple processes. However, parallel zarr writes are only optimal when
the chunk size are uniform. For me, they are not uniform---so, I
abandoned the idea of using zarr.
Currently, I am trying to use mpi4py to gather the data I want and save
it through a mother MPI process. I ran some test simulations and all was
fine (see attached script). However, when I use structural plasticity
with mpi4py I get an error that:
"nest.lib.hl_api_exceptions.DimensionMismatch: ('DimensionMismatch in
SetStatus_id: Expected dimension size: 1\nProvided dimension size: 20',
'DimensionMismatch', <SLILiteral: SetStatus_id>, ': Expected dimension
size: 1\nProvided dimension size: 20".
I believe that mpi4py somehow interferes with nest's internal mpi
mechanism. Perhaps it signals nest kernel to use 1 process rather than
the mentioned 20. If I remove the mpi4py import call, all works fine.
So, I cannot run the simulation using this mpi4py hack.
Do you notice something wrong with my approach? I'm just beginning to
use mpi4py, so I may be missing something here.
Alternatively, could you recommend a way to reduce the number of
exported files. For example, through zarr or other file formats which
support parallel writes.
I'd appreciate any suggestions. Thanks!
pyNEST version: 2.20.2
python version: Python 3.7.10
mpiexec/mpirun version: MPI Library for Linux* OS, Version 2021.3