Neuromodulation

Twenty-five Lessons from Computational Neuromodulation

Peter Dayan

Neural processing of brain is constrained by three problems. A) Computation is is radically distributed, yet amount of point-to-point connections are limited. (B) Bulk of these connections are structurally uniform, lacking target specific physical differences, (C) The brain's structure is relatively fixed, unable to explain the multifarious modalities under different conditions of neural ensemble.

Neuromulators address these problems by their multifarious and broad distribution, By enjoying and specialized receptor types in partially specific enough hey anatomical arrangements, and by their ability to mode activity and sensitivity of neurons and the strength of the plasticity of their synapses. 

As we should see later, architectural and neuromodulatory specifications are frequently integrated. We address Pavlovian (habitual) and model-based learning by reviewing acetycholine (Ach), dopamine (DA), norepinephrine (NE), and serotonin (5-HT). Phasic (transient) dopamine signal resulted from predictive cues provides a formal underpinning for the theory of incentive salience, which is concerned with motivational influences over the attention garnered by such stimuli. We also have noted substructure in the dopamine systems such as the mesocortical dopamine neurons that are excited rather than inhibited by punishment.

Goal-directed control, working memroy

Pavlovian mechanisms that lead to approach or withdrawal to external appetitive and aversive outcomes an predictors might inluence the way theenumeration works. States associated with reward could be more likely to be enomerated under the influence of dopamine and serotonin. Control over working memory can bave both instrumental and Pavlovian components. Basal ganglia could acquire policies that control the gating of information nito working memory using reinforcement learning. 

Uncertainty

The representation, updating and use of uncertainty have become major foci of computational treatments of neural information processing.

Learning

Once case of the link between ignorance and learning arises in the context of auto-associative memory models of the hippocampus. If an input is assessed to be novel, then it is stored; if the input is familiar, the recall process should remove noise from it and/or recall relevant context or associated information. The ability of neuromodulators to control the course of activity by regulating  pathways determines the activity of neurons is a common scheme.  ACh helps regulate oscillations that simultaneously affect multiple sub-regions of the hippocampal formation. It has been suggested that different pathways between theses regions are dominant at different phases of theta.

Model-based predictions and plans are dependent on learning about the structure of the environment and circumstances and outcome contingencies. This should be regulated by predictive uncertainty.

There is structure in a loop connecting cholinergic neuclei to sensory processing and prefrontal cortices.

There's evidence that tonic activity or levels of norepinephrine do indeed increase with predictable reversals in a simple reaction time task, and boosting no norepinephrine (NE) can speed the course of reversal learning. Reversals we shot popular way of inducing change, are normally signaled when actions and choices that used to be rewarded become unproductive or less productive; and actions that were formerly punished or nugatory become worthwhile. One might expect dopamine and serotonin to be involved directly in assessment and realization of reversals. Rapid change is normally the feature of a model based or goal-directed system. With NE, the projection of serotonin and dopamine to the striatum and prefrontal regions having implicated in form of behavioral flexibility such as reversal learning and set shifting.  Mechanism by which bonuses arise in model-based calculations is unknown. 

NE has long been linked with anxiety. NE helps to organize response in stress notably in conjunction with cortisol, a steroid hormone that acts as another neuromodulator. This involves everything from changing storage and usage via glucocorticoids to change the actual style of information processing.

Inference

Selective attention the impact of uncertainty should be governed by the utility associates with what can be discovered; one might expect that much of the inferential certainteed should be highly specific to circumstances of the task. There's evidence for involvement of both ACh and NE in controlling critical aspects of inference at both timescales mentioned above. Cholinergic neuromodulation ass also been implicated in aiding signal detection.

 For normal animals, ACh was substantially released over a short timescale on trials on which animals successfully detected the cue. At fine, sub-second timescale presentation of the cues the phasic release of ACh is related to expectation of the changing circumstances associated with the upcoming reward.

Discussion

For the first, we have seen common motifs such as heterogeneity in space (i.e. different receptor types with different affinities some localize on different systems) and heterogeneity in time ( with phasic and various timescales of tonic release) there's number forms of controls including self-regulation by autoreceptors, complex forms of inter-neuromodulator interaction. Also, loops between prefrontal areas and neuromodulator nuclei which exert mutual influence uptown each other. These, and indeed other functions of the neuromodulators, maybe complicated by corelease of other neurotransmitters and other neuromodulators through the same axons. 

Given the focus on decision-making, the key neuromodulators were dopamine, serotonin, acetylcholine, and norepinephrine, which you presented information about reward punishments and expected and unexpected uncertainty.

Concerning neural processing, the two most important systematic effects are controlling plasticity, and controlling whole pathways. Acetylcholine's influence on thalamocortical versus intracortical interactions. They are so dynamic effects, searches change the gain of competitive, decision-making circuits.

For future, measuring their local concentrations at target zones and selectively manipulate their activity or that of particular receptor types. 

BOLD signal my not would be very effective in helping understanding neuromodulation because the neuromodulators might be able to affect local blood flow directly themselves. There is much work to do to  understand overall networks and systems affects of changes that we know different neuromodulators lead to individual elements in those circuits. This may help understand the aspect of various sorts of heterogeneity. - e.g. What is achieved by the subtle differences within families of receptors and also the rich intertwining of neuromodulators. Model free and model based systems, with the former substantially based on neuromodulators such as dopamine and serotonin, whereas the latter depends on cortical processing.

My thoughts

The effects of neuromodulation is almost everywhere on every function of brains. They are the main mediator of signal processing across multiple spatial and temporal scales. In the review, the paper addressed four principal neuromodulators that implicated on model free, Pavlovian control, model-based, goal-directed control, uncertainty, inference, working memory, and learning. From signal transmission at synapses, to looping between striatum and cortex, their intertwining corelease and heterogeneous targets and receptors allows complex functions of brain be possible with such a limited, and fixed dendritic structures. This paper explained how the three limitations mentioned in the abstract are solved by neuromodulations.  

What do neuroscientists think of computational neural networks and deep learning?

As a student of neuroscience, I have never heard of computational neural networks. I guess you mean artificial neural networks. I can just speak for myself.

Many theorists believe there is a link between ANN, DL and brains, but to most neuroscientists only believe what evidence tell them. There is not much proof showing ANN and deep learning are how brains work.

The connection between ANNs and brains are nearly non-exsitent. The fact is that ANNs and DL only looks like real neural network from a layman’s perspective. There’s nothing they are alike when you scrutinize the details. For example, the ‘cell’ in ANNs stores values. However, in real cells there are just electrical potentials and the information are full of noises. Besides, the connections in our brains are quite random, and full of recurrent loops in different levels, while in DL the network structures are feed-forward and fixed. In another words, it seems the more successful the ANNs, the less it resembles real neural networks. Even many vision scientists use probabilistic models for visual cortex, it is not explained how a global cost-function minimization, back propagation are achieved neurologically. For deep learning such as deep belief networks, the training is not on-line but off-line (batch training, 2 phase training), which is also very different from our brains.

On the other hand, neuroscientists have fairly clear idea of how grid cells and place cells in hippocampus work to encode spacial memory based on experiments, and the machenism is very different from ANNs. Other successful models for cerebellum are very different from ANNs, either. If ANNs and DL are generic models of brains, then the mechanisms should not be so different.

Deep Robotic Learning Sergey Levine

Deep Robotic Learning
Sergey Levine

Emergence of Counsicousness and Attention.

There are only two kinds of things that we need to pay attention. First kind of thing is the thing you want to do, but you can't do. The second kind of thing is the thing you can do, but you don’t want to do. Apart from them, things just pass by like fog and smoke and we don’t see exactly how and when they were done. They are things that we do everyday unconsciously. They are routines and trifles that we’ve done thousands of times that some sort of muscle autonomy just takes over, free us from the tedious labor. Eating while watching TV, talking on a cellphone while driving, Reading papers in the toilet, or listening to music while doing homework. We seem to avoid paying any attention to those tedious and trivial jobs as much as we can even when we nothing better to do. These tasks are duties which we have no choice but doing , otherwise our bodies will stop operating. As long as it agrees with our nature, and nothing goes against it, then our bodies just know how and when to do it right itself. Put it philosophically, our nature conforms nature itself. It follows the laws of thermal dynamics, and try to maximize entropy if we don’t spend energy to stop it.

Attention is inhibition in cognitive sense. We have to inhibit our desires when we can’t do it. We can’t spend freely, wake up late, or smoke as much as we want , even though it’s what life is about. We must inhibit the desires when our knowledge tells us they are detrimental. Therefore, we are painfully paying attention not letting our nature unleashed. On the other hand, we are forced to do things we hate and never get used to on daily bases. We can choose the easy way, but we don’t. because our knowledge tell us they are the necessary vice, otherwise something worse may happen. We have to inhibit our disinclination every second to fight our natural aversion for the longer good.

Inhibition deforms our mental structure, builds up mental tension, stress, and strain, and transforms entropy into potential energy in the mental deformation. Attention is inhibition in phenomenological sense, it is heat and light emitted when potential energy stored in deformation exceeds the energy threshold of the structure. The deformation becomes fracture, and permanently changes the mental default structure. Mental energy discharge is attention, attention leads to mental suffer, mental suffer leads to consciousness, consciousness leads to selfness. Selfness is the crystallization of entropy minimization, the maximization of inhibition, and the singularity of universe.

Inhibition is the essential element of nervous system. It is impossible for nervous system to generate meaningful patterns without incorporating inhibitory signal. Without inhibitory neurons, excitation neurons will fire unbridled, and we will be led to delirious madness and eventual self-destruction. Inhibitory neurons are the miniature self. They defines the character of nervous systems. They are the source of animation, and the starting point of the betrayal of God.

Training Deep NN and Theory of Sleep

It is just an idea, and probably somebody has already been working on it. I still feel the urge to share with you about how I think the Deep NN has inspired me on the evolutional purpose of dreaming, and how theory of dreaming can help us better train Deep NN. However, I must disclaim here, that the following discourse are purely speculative.

NN has been known as a kind of unsupervised learning. i.e., it learns the regularity of data without the help of human expert to label them. The NN then can be trained via ‘backprapagation’, during which the weights of the links, and the value (activity) of the nodes and be updated via optimization of ‘objective function’. Typically, cross entropy is used as objective function. The typical problem one usually confront when training the network is the gradient descent on the landscape of the objective function may fall into local minimum. Several techniques are widely used to help escape from suboptimum solution, such as annealing, adding momentum term, or adding stochastic factor into the training. Here I propose an idea that is inspired by damped oscillation pattern of sleep. The damped oscillation of sleeping stage consists four stages, each of which lasts from 10 to 20 minutes with slight variance depending on individual and various physiological factors. The oscillation is illustrated in the following graph.

Each stage are reported to be responsible for different purpose. The REM -> Stage 4 section can be think for as backward propagation, while the Stage 4 -> REM section represents the forward propagation. The shallow-deep sleep cycle help strengthen the memory and also help internalize the vast amount of transient episodes into deep, invariant knowledges. I will elaborate in the next paragraph. The picture below shows the famous LeNet proposed by Yann LeCun. The net resembles brain structure not only in terms of topology but how the information is processed. Each layer ‘pools’(samples) small patch of the input and forms a feature space that is going to be sampled by a super-layer.You can see the trained middle layers as filters of information. Finally, the every single nodes in the end-layer can distinguish objects exclusively. In broader sense, the deep NN can condense knowledge from a huge batch of unsorted data, be it image, sounds, or movements. The success of Deep NN sheds light on the long coveted universal theory that explains how might brain learns, predict, interpret, and create things. But still, we are very far from finding such a theory, for the anatomical counterparts and physiological mechanism that justifies the NN are not yet approved. Even more, I am neither saying that Deep NN can explains EVERYTHING of brain mechanism nor the universal theory exists.

In general it is agreed upon that sleep is essential for long term memory consolidation, and can enhance cognitive processing. The REM sleep, where we usually dream and can easily be awaken by external disturbance, seems to be playing a critical role of learning. It is instructive that during dreaming our mental activity is actually quite similar to the waking state, and the major difference is the mental process is isolated from body movement. By analyzing the texture of dream, we will find out , unsuprisingly in retrospect, that the function and form conforms each other. The structure of dreams reflects exactly how we stores invariances in the deep, hierarchical structure of filters of different functional level.

In the first cycle of sleep, we dive into deep sleep (stage 1 to stage 4) from aware state. This process correspond to the back propagating phase of LeNet (Input layer to end layer), where in each stage of sleep corresponds to optimization of weights of each layer of NN. It doesn’t mean that the physical counterpart of Deep NN must have four layers, because the definition of stage is artificial and only serve certain purpose. There are not real distinct boundaries between them, it is more like a gradual process. During this half-cycle, the optimization of weights cannot be too good, since it can easily fall into local minimum.

At the deep sleep, end-layer that interfaced with hormone signals produces by glands[1], and other subcortical bodies [2] received instruction and starts to send signal back to sub-layers. These end-layers represents emotional components, and inverse the previous process, starting to predict the possible outcome in the sub feature space given such emotions.Since the inverse problem does not exist a unique solution, we won’t see a playback of our input, but a physically and causally plausible theatre, in which a unpredictable, bizarre, but animated drama is on the show. When we see such a show during sleep, our other parts of brain are aroused, resulting awake-like brain state. The vivid experience are overall endogenous, but its constituents can be novel to each other perceptrons, as the hierarchical brain structure always branch out when goes downward, and the genesis of artifacts is nonlinear due to convoluted operation or quasi-randomized internal driving sources.

During the first REM stage we are exposed to fabricated scenarios that appears arbitrary and illogical, but are strongly emotional and animated. These scenarios are oftentimes consistent in sense of causality and physics, but their motifs and development are totally unpredictable. Take the visual recognition for example. We can generate a face by synthesize features according to the structure of NN, but no single line on of the synthesized image will have the same shape, same shade, and orientation. In a sense the dream provides artificial data let allows us to train the Deep NN again. The subsequent descent of the sleep stage allows optimization for the second time. This helps the NN escape from local optimum of objective function.

This cycle of forward modeling -backpropagation is repeated a few times but not to many for 1) the data novelty deprecate after each cycle. 2) over-fitting. 3) time to wake up. Due to the pyramid shaped hierarchical structure, deeper layers has progressively less perceptrons/ filters, and therefore more prone to over-fitting. This accounts for why the depth of each sleep cycle decrease in light of learning and memory. The lower the hierarchy, the more perceptrons to be trains, and therefore the shallower stages (REM, stage 1, stage 2) occupy the primary proportion in the later part of the whole sleep.

What is the benefit of interfacing the emotional layer with emotional related hormone sources ? It is postulated that the hormones are supervising signal that instruct our brain to learned scenarios that embeds negative emotions ( stress, anxiety, fear, and anger) so that we can handle social relations better. Learning to endure and even utilize our negative emotions during sleep prepares us better to handle frustration and unpredictable difficulties in real world, where most of time we live under certain pressure of survival. It equips us with a inherit sense of crisis, which helped our ancestors survived disasters that those with different mindsets did not.

Footnote:

[1]such as pineal gland, Pituitary gland

[2] like amygdala (emotional memory), mammillary body (recollective, or episodic memory), and hippocampus (spacial memory, STM, LTM).

[3] Other theories of sleep : Activation synthetic theory ; Continual -activation theory ; Reverse learning ; Dreams as excitations of long term memory;