Reinforcement learning of spiking neural networks using trace variables for synaptic weights with memristive plasticity

Impulse neural networks, suitable for hardware implementation based on memristors, are very promising for robotics due to their energy efficiency. However, reinforcement learning algorithms using such networks remain poorly understood. One of the key motivations for using memristors as network weights is, in addition to energy efficiency, their ability to learn (change conductivity) in real time by superimposing voltage pulses from pre- and postsynaptic signals. The article presents the results of numerical modeling of a spiking neural network (SNN) with memristive synaptic connections, which approximately solves the optimal control problem using trace variables for weight changes, allowing one to approach reinforcement learning on a true time scale. The fundamental possibility of such training in the task of holding a pole on a moving platform is shown, a comparison of various reward functions is given, and assumptions are made about ways to increase the effectiveness of this approach.