According to age, a person needs 7 to 13 hours of complete sleep in 24 hours a day. During the process of sleeping, many things happen in the body, including slow heartbeat and breathing, slowing down of the body’s metabolism and regulation of hormone levels. In such a state, the body is completely at peace, but in the meaime, the brain takes a differe behavior.
The brain is very busy during sleep because it repeats what we have learned during the day in order to learn better. In fact, sleep helps to reorganize information and memories in order to prese them in the most efficie way possible when awake. It is ieresting to know that the same procedure is true for artificial neural networks, these strange human creatures!
Catastrophic forgetting in artificial neural networks
Artificial neural networks use an architecture similar to the human brain to improve numerous technologies and systems, from basic science and medicine to finance and social networking. In this regard, it is ieresting to know that although in some ways, these human creatures have achieved superhuman functions, such as very high computing speed, but they have failed in one key aspect. When artificial neural networks learn things sequeially, rewriting new information on previous information, such a phenomenon is called “Catastrophic Forgetting”.
When artificial neural networks learn things sequeially, they overwrite new information on top of previous information, which has led to a kind of failure in the developme of these systems.
In corast to artificial neural networks, the human brain is constaly learning, incorporating new data alongside existing knowledge. In this learning style, when new teachings are combined with regular periods of sleep, it helps to consolidate information better and learn more deeply. So on November 18, 2022, computational biologists are looking further io how biological models can help reduce the risk of catastrophic forgetting in artificial neural networks and increase their applicability in a wide range of research fields.
Sleeping helps neural networks learn better
In this regard, scieists used special neural networks that artificially imitate the behavior of natural nervous systems. In fact, in this new approach, instead of the information being constaly transferred and the eire system constaly communicating, the information is transferred as discrete eves (clustered) at certain times.
Through this procedure, the scieists found that when these clustered networks learn new information in the form of occasional, non-coinuous periods (similar to sleep), the rate of catastrophic forgetting is reduced. According to the authors of the study, like the human brain, sleep allows neural networks to recall old memories without explicitly using old training data. In the human brain, memories are represeed by patterns called synaptic weight, which means the strength or range of communication between two neurons.
Catastrophic forgetting rates are reduced when cluster networks learn new information in the form of occasional, non-coinuous periods (similar to sleep).
Basically, when we are learning new information, neurons light up in a certain order, and this issue is also a factor for increasing synapses among differe neurons. During sleep, these same synaptic patterns (which occur in the awake state) repeat themselves, which is called reactivation or replay.
The small space between two neurons is called a synapse, through which the nerve impulse is transmitted by neurotransmitters from the transmitter to the receiver.
Synaptic plasticity, which includes the capacity to change or reshape connections, is also prese during sleep and can reinforce synaptic weight patterns that represe memory and help preve forgetting or rewriting information on top of new information. The researchers of this study applied this approach to artificial neural networks and found that such a procedure helps these networks avoid catastrophic forgetting. The result of this research means that these networks can coinuously learn new information just like humans or animals.
In the end, it should be noted that understanding how the human brain processes information during sleep, which can help strengthen memory, is one of the areas of study that can lead to significa progress in the improveme of artificial neural networks.
Source: Science Daily
