Hi Xavier,

I have now created a PR that should fix the problem with hanging simulations. You can find it at

https://github.com/nest/nest-simulator/pull/3101

I’d much appreciate it if you could check that it solves your problem.

Best,

Hans Ekkehard

-- 

Prof. Dr. Hans Ekkehard Plesser

Department of Data Science

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

From: Hans Ekkehard Plesser <hans.ekkehard.plesser@nmbu.no>
Date: Sunday, 11 February 2024 at 16:46
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] Re: Weights assignment error with mpirun

Hi Xavier,

I think I understand your MPI-hang problems now (see comment in the issue) and can suggest a work-around

        if (w.size>1):
                w_normed = w / sum(abs(w))  # L1-norm
                w_nparray = 2. * w_normed
                conn.weight = w_nparray

       else:

             conn.weight = []

The important thing is that the SynapseCollection.set() method, which is called by the assignment, is executed by all MPI ranks, even if their conn is empty. Assigning to an empty conn just does nothing, except MPI communicate with the other ranks.

The other work around is to call SetStatus(), which avoids the MPI calls altogether and thus is most likely more efficient.

Best,

Hans Ekkehard

-- 

Prof. Dr. Hans Ekkehard Plesser

Department of Data Science

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

From: Xavier Otazu <xotazu@cvc.uab.cat>
Date: Friday, 9 February 2024 at 14:00
To: users@nest-simulator.org <users@nest-simulator.org>
Subject: [NEST Users] Re: Weights assignment error with mpirun

[Du mottar ikke ofte e-post fra xotazu@cvc.uab.cat. Finn ut hvorfor dette er viktig på https://aka.ms/LearnAboutSenderIdentification ]

Hans,

Inhomogeneous spike generator is a very nice solution ... Thanks for the tip! I didn't thought about it :-) I will try it (although at this moment my main time consuming process is the normalization step, not the Simulate() preparation time :-/ )

Xavier

Hans Ekkehard Plesser wrote:
> Hi Xavier,
>
> If you know the time course of you poisson process rates beforehand, you can use the
> inhomogeneous_poisson_generator, which changes to new rates at given time points.
>
> The rate calculation of the poisson generator is precisely on of the cases where SetStatus
> inside a RM context causes problems. This will not change, because one of the key points
> about the RM context is to avoid time overhead for neuron configuration.
>
> Best,
> Hans Ekkehard
>
> --
>
> Prof. Dr. Hans Ekkehard Plesser
>
> Department of Data Science
> Faculty of Science and Technology
> Norwegian University of Life Sciences
> PO Box 5003, 1432 Aas, Norway
>
> Phone +47 6723 1560
> Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no>
> Home http://arken.nmbu.no/~plesser
>
>
>
> From: Xavier Otazu &lt;xotazu(a)cvc.uab.cat&gt;
> Date: Friday, 9 February 2024 at 11:44
> To: users(a)nest-simulator.org &lt;users(a)nest-simulator.org&gt;
> Subject: [NEST Users] Re: Weights assignment error with mpirun
> [Du mottar ikke ofte e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette er viktig på
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi Hans,
>
> Inside an RM context, I am not only changing synaptic weights, but also Poisson spike
> generators rates. At every loop iteration I change the "input image", and I
> simulate it using a spike generator. I tried an RM context some time ago (previous NEST
> versions) and I realized that rate didn't change (I guess because the non-updated
> internal variables). Is this behaviour changed on 3.6 version?
>
> Related to the neuroscientific question I understand you ask me about the temporal
> evolution of weights when externally modifying them. When I just used STDP, my weights
> grow indefinitely (a well known problem) or just saturates to W_max. Since in many models
> input weights are normalized, I applied normalization to avoid this growing problem (in
> addition, there are some biophyisical studies showing that this kind of normalization
> exists in biological systems). Another question is the "continuity" (first
> derivative) of temporal evolution of weights ... the answer is "I do not know"
> :-/
>
> Best,
> Xavier
>
> Hans Ekkehard Plesser wrote:
> >   Hi Xavier,
> >
> >  Actually, in your case the Prepare-Run-Cleanup approach should work, since you use
> >  SetStatus only on synaptic weights. The reason we warn about unpredictable behaviour in
> >  case of SetStatus inside a RunManager context is that Prepare() calls a pre_run_hook()
> > on
> >  all network nodes to allow them to precompute internal variables. So if you changed
> > neuron
> >  parameters inside an RM context, the nodes would update with an inconsistent set of
> >  externally set parameter values and internally not-updated variables. Synapses do not
> > have
> >  such a pre-run-hook, so there you should be way safe. Note that you cannot
> >  Connect/Disconnect in a RM context, since the connection infrastructure needs to be
> >  updated by Prepare().
> >
> >  Finally, one could wonder about how the external manipulation of the synaptic weights
> >  interacts with the internal evolution of the weights through STDP, i.e., what the actual
> >  time-evolution equations for your weights become when taking normalization into account.
> >  But that is a neuroscientific question, not a NEST-technical one.
> >
> >  Best,
> >  Hans Ekkehard
> >
> >  --
> >
> >  Prof. Dr. Hans Ekkehard Plesser
> >
> >  Department of Data Science
> >  Faculty of Science and Technology
> >  Norwegian University of Life Sciences
> >  PO Box 5003, 1432 Aas, Norway
> >
> >  Phone +47 6723 1560
> >  Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no>
> >  Home http://arken.nmbu.no/~plesser
> >
> >
> >
> >  From: Xavier Otazu &lt;xotazu(a)cvc.uab.cat&gt;
> >  Date: Thursday, 8 February 2024 at 21:42
> >  To: users(a)nest-simulator.org &lt;users(a)nest-simulator.org&gt;
> >  Subject: [NEST Users] Re: Weights assignment error with mpirun
> >  [Du mottar ikke ofte e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette er viktig på
> >  https://aka.ms/LearnAboutSenderIdentification ]
> >
> >  Hi Hans,
> >
> >  Great news to be able to call GetConnections() just once ... it speedsA LOT my runnings
> >  (from 230-ish seconds to 90-ish) :-)
> >
> >  I tried the Prepare/Run/Cleanup some time ago (nice solution), but since I have to use
> >  SetStatus() to update normalized weights between Prepare() and Cleanup(), I cannot use
> > it
> >  :-(
> >
> >  Best,
> >  Xavier
> >
> >  Hans Ekkehard Plesser wrote:
> >     Hi Xavier,
> >
> >   The result of `GetConnections()` remains valid as long as no new synapses are added or
> >   synapses deleted, so in your case, you only need to call it once.
> >   When you then do a `conn.weight`, you will get the weights at that point in time. But
> >  note
> >   that NEST only updates synaptic weights when a spike actually passes through the
> >  synapse,
> >   so you will get for each synapse the weight as it was at the end of the plasticity
> >  update
> >   when the most recent spike passed through that synapse.
> >
> >   Instead of calling Simulate() repeatedly, you can also use a RunManager context for
> >  higher
> >   efficiency, see
> >   https://nest-simulator.readthedocs.io/en/stable/nest_behavior/running_simul….
> >
> >   In case you should be wondering why global instantaneous normalization is so cumbersome
> >  in
> >   NEST: We have focused on support for biologically plausible networks so far and
> >   instantaneous global normalization is just not biologically plausible (nor physically
> >   feasible with a view to causality).
> >
> >   Best,
> >   Hans Ekkehard
> >
> >   --
> >
> >   Prof. Dr. Hans Ekkehard Plesser
> >
> >   Department of Data Science
> >   Faculty of Science and Technology
> >   Norwegian University of Life Sciences
> >   PO Box 5003, 1432 Aas, Norway
> >
> >   Phone +47 6723 1560
> >   Email hans.ekkehard.plesser@nmbu.no<mailto:hans.ekkehard.plesser@nmbu.no>
> >   Home http://arken.nmbu.no/~plesser
> >
> >
> >
> >   From: Xavier Otazu &lt;xotazu(a)cvc.uab.cat&gt;
> >   Date: Thursday, 8 February 2024 at 19:46
> >   To: users(a)nest-simulator.org &lt;users(a)nest-simulator.org&gt;
> >   Subject: [NEST Users] Re: Weights assignment error with mpirun
> >   [Du mottar ikke ofte e-post fra xotazu(a)cvc.uab.cat. Finn ut hvorfor dette er viktig
> > p?
> >   https://aka.ms/LearnAboutSenderIdentification ]
> >
> >   Hi Hans,
> >
> >   Thanks for your information. Fortunately, I do not normalize output connections (only
> >   input), hence, this code works for me.
> >
> >   Another question. Take that I call nest.Simulate() inside a for loop and I want to
> >   normalize weights after every iteration. In order to avoid calling GetConnections() for
> >   every loop iteration, may I get the connections calling GetConnections() just once? May
> >  I
> >   safely suppose that after every Simulate() new weights of my connections (they are
> >  changed
> >   because I use STDP) will be available (for reading) using the same variable? An example
> >   code would be something like:
> >
> >   my_conn = GetConnections(...)
> >
> >   for iter in range(...)
> >      nest.Simulate(...)
> >      normalize_connections(my_conn) # function that normalizes weights and calls
> >   nest.SetStatus(my_conn, ...)
> >
> >
> >   Many thanks in advance!
> >
> >   Xavier
> >   _______________________________________________
> >   NEST Users mailing list -- users(a)nest-simulator.org
> >   To unsubscribe send an email to users-leave(a)nest-simulator.org
> >  _______________________________________________  NEST Users mailing list --
> > users(a)nest-simulator.org
> >  To unsubscribe send an email to users-leave(a)nest-simulator.org
> > _______________________________________________
> NEST Users mailing list -- users(a)nest-simulator.org
> To unsubscribe send an email to users-leave(a)nest-simulator.org
Hans,
_______________________________________________
NEST Users mailing list -- users@nest-simulator.org
To unsubscribe send an email to users-leave@nest-simulator.org