Artificial neural networks sleep for better learning

According to age, a person needs 7 to 13 hours of complete sleep in 24 hours a day. During the process of sleeping, many things happen in the body, including slow heartbeat and breathing, slowing down of the body’s metabolism and regulation of hormone levels. In such a state, the body is completely at peace, but in the meantime, the brain takes a different behavior.

The brain is very busy during sleep because it repeats what we have learned during the day in order to learn better. In fact, sleep helps to reorganize information and memories in order to present them in the most efficient way possible when awake. It is interesting to know that the same procedure is true for artificial neural networks, these strange human creatures!

Catastrophic forgetting in artificial neural networks

Artificial neural networks use an architecture similar to the human brain to improve numerous technologies and systems, from basic science and medicine to finance and social networking. In this regard, it is interesting to know that although in some ways, these human creatures have achieved superhuman functions, such as very high computing speed, but they have failed in one key aspect. When artificial neural networks learn things sequentially, rewriting new information on previous information, such a phenomenon is called “Catastrophic Forgetting”.

When artificial neural networks learn things sequentially, they overwrite new information on top of previous information, which has led to a kind of failure in the development of these systems.

In contrast to artificial neural networks, the human brain is constantly learning, incorporating new data alongside existing knowledge. In this learning style, when new teachings are combined with regular periods of sleep, it helps to consolidate information better and learn more deeply. So on November 18, 2022, computational biologists are looking further into how biological models can help reduce the risk of catastrophic forgetting in artificial neural networks and increase their applicability in a wide range of research fields.

Sleeping helps neural networks learn better

In this regard, scientists used special neural networks that artificially imitate the behavior of natural nervous systems. In fact, in this new approach, instead of the information being constantly transferred and the entire system constantly communicating, the information is transferred as discrete events (clustered) at certain times.

Through this procedure, the scientists found that when these clustered networks learn new information in the form of occasional, non-continuous periods (similar to sleep), the rate of catastrophic forgetting is reduced. According to the authors of the study, like the human brain, sleep allows neural networks to recall old memories without explicitly using old training data. In the human brain, memories are represented by patterns called synaptic weight, which means the strength or range of communication between two neurons.

Catastrophic forgetting rates are reduced when cluster networks learn new information in the form of occasional, non-continuous periods (similar to sleep).

Basically, when we are learning new information, neurons light up in a certain order, and this issue is also a factor for increasing synapses among different neurons. During sleep, these same synaptic patterns (which occur in the awake state) repeat themselves, which is called reactivation or replay.

The small space between two neurons is called a synapse, through which the nerve impulse is transmitted by neurotransmitters from the transmitter to the receiver.

Synaptic plasticity, which includes the capacity to change or reshape connections, is also present during sleep and can reinforce synaptic weight patterns that represent memory and help prevent forgetting or rewriting information on top of new information. The researchers of this study applied this approach to artificial neural networks and found that such a procedure helps these networks avoid catastrophic forgetting. The result of this research means that these networks can continuously learn new information just like humans or animals.

In the end, it should be noted that understanding how the human brain processes information during sleep, which can help strengthen memory, is one of the areas of study that can lead to significant progress in the improvement of artificial neural networks.

Source: Science Daily