#reinforcement learning

[compneuropapers]

Interesting Papers for Week 46, 2025

Embedding stochastic dynamics of the environment in spontaneous activity by prediction-based plasticity. Asabuki, T., & Clopath, C. (2025). eLife, 13, e95243.

Predictive learning rules generate a cortical-like replay of probabilistic sensory experiences. Asabuki, T., & Fukai, T. (2025). eLife, 13, e92712.

Awake reactivation of cortical memory traces predicts subsequent memory retrieval. Duan, W., Lu, P., Xu, Z., Wang, J., Lu, Y., Wang, M., Paller, K. A., Axmacher, N., & Wang, L. (2025). Progress in Neurobiology, 250, 102778.

Decision Threshold Learning in the Basal Ganglia for Multiple Alternatives. Griffith, T., Baker, S.-A., & Lepora, N. F. (2025). Neural Computation, 37(7), 1256–1287.

Unveiling the role of local metabolic constraints on the structure and activity of spiking neural networks. Jaras, I., Orchard, M. E., Maldonado, P. E., & Vergara, R. C. (2025). PLOS Computational Biology, 21(6), e1013148.

Sleep need–dependent plasticity of a thalamic circuit promotes homeostatic recovery sleep. Lee, S. S., Liu, Q., Cheng, A. H. R., Kim, D. W., Boudreau, D. M., Mehta, A., Keles, M. F., Fejfer, R., Palmer, I., Park, K. H., Münzberg, H., Harris, T. D., Graves, A. R., Blackshaw, S., & Wu, M. N. (2025). Science, 388(6753).

Multi-timescale reinforcement learning in the brain. Masset, P., Tano, P., Kim, H. R., Malik, A. N., Pouget, A., & Uchida, N. (2025). Nature, 642(8068), 682–690.

A common alteration in effort-based decision-making in apathy, anhedonia, and late circadian rhythm. Mehrhof, S. Z., & Nord, C. L. (2025). eLife, 13, e96803.

Dynamics of energy-efficient coding in visual cortex. Moosavi, S. A., Pastor, A., Ornelas, A. G., Tring, E., & Ringach, D. L. (2025). Journal of Neurophysiology, 133(6), 2006–2013.

Distal tuft dendrites predict properties of new hippocampal place fields. O’Hare, J. K., Wang, J., Shala, M. D., Polleux, F., & Losonczy, A. (2025). Neuron, 113(12), 1969-1982.e7.

Neural dynamics encoding risky choices during deliberation reveal separate choice subspaces. Peters, L. M., Roadarmel, A., Overton, J. A., Stickle, M. P., Kong, Z., Saez, I., & Moxon, K. A. (2025). Progress in Neurobiology, 250, 102776.

Embodied decisions as active inference. Priorelli, M., Stoianov, I. P., & Pezzulo, G. (2025). PLOS Computational Biology, 21(6), e1013180.

Predicting neuronal firing from calcium imaging using a control theoretic approach. Rondoni, N. A., Lu, F., Turner-Evans, D. B., & Gomez, M. (2025). PLOS Computational Biology, 21(6), e1012603.

Hippocampal Sharp-Wave Ripples Decrease during Physical Actions Including Consummatory Behavior in Immobile Rodents. Sakairi, T., Kawabata, M., Rios, A., Sakai, Y., & Isomura, Y. (2025). Journal of Neuroscience, 45(25), e0080252025.

Cholinergic modulation of neural networks supports sequential and complementary roles for NREM and REM states in memory consolidation. Satchell, M., Butel-Fry, E., Noureddine, Z., Simmons, A., Ognjanovski, N., Aton, S. J., & Zochowski, M. R. (2025). PLOS Computational Biology, 21(6), e1013097.

Precise spike-timing information in the brainstem is well aligned with the needs of communication and the perception of environmental sounds. Scholes, C., Coombes, S., Palmer, A. R., Rhode, W. S., Mill, R., & Sumner, C. J. (2025). PLOS Biology, 23(6), e3003213.

A multidimensional distributional map of future reward in dopamine neurons. Sousa, M., Bujalski, P., Cruz, B. F., Louie, K., McNamee, D. C., & Paton, J. J. (2025). Nature, 642(8068), 691–699.

Cholinergic feedback for modality- and context-specific modulation of sensory representations. Yu, B., Yue, Y., Ren, C., Yun, R., Lim, B., & Komiyama, T. (2025). Science, 388(6753), 1324–1329.

Distinct neural population code and causal roles of primate caudate nucleus in multimodal decision-making. Zeng, Z., Zhang, C., Xu, Y., He, H., & Gu, Y. (2025). Nature Communications, 16, 5253.

A neural mechanism for learning from delayed postingestive feedback. Zimmerman, C. A., Bolkan, S. S., Pan-Vazquez, A., Wu, B., Keppler, E. F., Meares-Garcia, J. B., Guthman, E. M., Fetcho, R. N., McMannon, B., Lee, J., Hoag, A. T., Lynch, L. A., Janarthanan, S. R., López Luna, J. F., Bondy, A. G., Falkner, A. L., Wang, S. S.-H., & Witten, I. B. (2025). Nature, 642(8068), 700–709.

[choconado]

small brain moment here, but how is markov property justified. my tomorrow's actions were dependent on my today;s actions but not on my yesterdays actions

BUT my today;s actions were dependent on my yesterdays action and not before yesterdays actions

so doesnt this technically mean that there IS a chain

tomorrow is dependent on TODAY and today is dependent on YESTERDAY so tomorrow is also technically dependent on yesterday???

[neil-minyard-josten]

I have a feeling Neil Josten would love learning about Reinforcement learning.

[datascienceunicorn]

Episode 1 of DataCafé is all about Multi-Armed Bandits

🎰🎰🎰

Catch it when it airs Monday! 👂

[gendergreen]

An Autistic Psych Major's rant on learning about Conditioning.

I'm a graduating senior. I've taken multiple psych classes. The majority of them have a review of a fairly fundamental concept, conditioning.

As a scholar, this is a pretty fundamental concept of psychology and it's history. Pavlov Skinner are key figures. And that's all fine and good.

That being said, I'm a survivor of ABA. Consider this your trigger warning.

ABA has roots in B.F. Skinner's operant conditioning where he shocked rats to demonstrate punishment as a negative reinforcer and reward as a positive reinforcer.

B.F. Skinner's work directly influenced Ivar Lovaas who basically created ABA. This motherfucker used physical punishment against autistic children because they weren't acting neurotypical enough for him. (negative operant conditioning)

ABA today doesn't do that since that's obviously unethical. When I was given ABA, I was rewarded for my neurotypical behaviors with reinforcers with fucking sticker charts, snacks, and fucking agency to do what I wanted. (Eventually, you just comply. Token economies do not work, they just coerce compliance into neurotypical behavior, which is deeply damaging to self-esteem.)

Cue this last quarter.

My autistic-like behaviors were "redirected". At it's tamest, it'd be ignoring or a gentle nudge from the "behavior technician". (Yes. You heard me right. Behavior Technician.) Other negative reinforcers includes denial of food, being isolated from peers, and deprivation of agency. At it's most extreme, I was forcibly marched with a finger in my belt loop and hand on my back and pushed into a small room (the size of a medium-large walk-in closet), where I would be trapped in by the therapist, despite my screams to let me out.

So, yeah, almost every psych class I've had, I've had to brace myself for a brief lecture on conditioning. Listen, professor, I don't need to hear this lecture. I. FUCKING. LIVED. IT. I could fucking teach this. It's been so in-depth and bad before that I've gotten up and stepped out of the room.

Cue my 300-level cognitive class. It's online. We have to watch the "review" lectures from his 100-level intro class. I'm not listening to this. Fuck, just looking at a slide on reinforcers and punishments resulted in me closing my laptop and hyperventilating. I don't need to watch this review.

Unfortunately, we're learning about more advanced Classical Conditioning stuff that I can't bullshit, since I don't know that. (Kamin Blocking Effect, Rescorla-Wagner Model, that stuff.) Pavlovian conditioning still is better than Skinner conditioning. But, FUCK. Conditioning lectures are still conditioning lectures, and this could be way worse, but fuck this triggering bullshit.

Listen, I get that this is foundational cognitive shit. But this isn't fucking fair. Fuck Skinner. Fuck Lovaas. Fuck McEachin and Leaf. Fuck all of them for their fucked up contributions to this field and for making me hate an aspect of a major I love. Fuck the fact that I'm neurotypical passing enough and I'm not about to lose social capital by speaking up. Fuck ABA. Fuck conditioning.

Why do I have to be triggered by MULTIPLE classes because some assholes thought it was okay to demand compliance out of autistic children and then literally traumatized us? A staggering amount of us who are survivors of ABA have PTSD. And I'm so glad I'm fucking graduating because hopefully this is the last time I have to deal with this fucking bullshit.

*drops mic*

[michael-langdon-appreciation]

Y/N tugs on Michael's hair a bit too hard and he tells her to be gentle and then she does it like 2 more times and the 4th time she does it he completly stops moving his hips and his demon face comes out and he just quietly says 'gently' and then ruthlessly thrusts into her with demon face

OMG this is exactly the kinda scenario where this would happen!!! 

I love the idea of him using it for reinforcement, even just to get a kick out of it. You wanna steal a chip off his plate - better check with the demon face first! One day Michael went into the Cooperative with it just to assert himself as someone you do not fuck with. 

[imanythingbut0rdinary]

The latest Tweets from Chelsea 🦄 (@CJBrok3nWarri0r). Old acc.locked.👎 🏴󠁧󠁢󠁷󠁬󠁳22. @ddlovato 💜 Coffee, Fashion, Music, Marvel. Trying to recov

So my old account has been locked for a reason I don't know why. 😢I'm distraught it took me years to build. I am appealing but no answer yet. 👎

Please follow my new account to help me build back up.

TIA

[vespaengine]

Serving article comments using reinforcement learning of a neural net

Don’t look at the comments. When you allow users to make comments on your content pages you face the problem that not all of them are worth showing — a difficult problem to solve, hence the saying. In this article I’ll show how this problem has been attacked using reinforcement learning at serving time on Yahoo content sites, using the Vespa open source platform to create a scalable production solution.

Yahoo properties such as Yahoo Finance, News and Sports allow users to comment on the articles, similar to many other apps and websites. To support this the team needed a system that can add, find, count and serve comments at scale in real time. Not all comments are equally as interesting or relevant though, and some articles can have hundreds of thousands of comments, so a good commenting system must also choose the right comments among these to show to users viewing the article. To accomplish this, the system must observe what users are doing and learn how to pick comments that are interesting.

Here I’ll explain how this problem was solved for Yahoo properties by using Vespa — the open source big data serving engine. I’ll start with the basics and then show how comment selection using a neural net and reinforcement learning was implemented.

Real-time comment serving

As mentioned, the team needed a system that can add, find, count, and serve comments at scale in real time. The team chose Vespa, the open big data serving engine for this, as it supports both such basic serving as well as incorporating machine learning at serving time (which we’ll get to below). By storing each comment as a separate document in Vespa, containing the ID of the article commented upon, the ID of the user commenting, various comment metadata, and the comment text itself, the team could issue queries to quickly retrieve the comments on a given article for display, or to show a comment count next to the article:

In addition, this document structure allowed less-used operations such as showing all the articles of a given user and similar.

The Vespa instance used at Yahoo for this store about a billion comments at any time, serve about 12.000 queries per second, and about twice as many writes (new comments + comment metadata updates). Average latency for queries is about 4 ms, and write latency roughly 1 ms. Nodes are organized in two tiers as a single Vespa application: A single stateless cluster handling incoming queries and writes, and a content cluster storing the comments, maintaining indexes and executing the distributed part of queries in parallel. In total, 32 stateless and 96 stateful nodes are spread over 5 regional data centers. Data is automatically sharded by Vespa in each datacenter, in 6–12 shards depending on the traffic patterns of that region.

Ranking comments

Some articles on Yahoo pages have a very large number of comments — up to hundreds of thousands are not uncommon, and no user is going to read all of them. Therefore it is necessary to pick the best comments to show each time someone views an article. Vespa does this by finding all the comments for the article, computing a score for each, and picking the comments with the best scores to show to the user. This process is called ranking. By configuring the function to compute for each comment as a ranking expression in Vespa, the engine will compute it locally on each data partition in parallel during query execution. This allows executing these queries with low latency and ensures that more comments can be handled by adding more content nodes, without causing an increase in latency.

The input to the ranking function is features which are typically stored in the document (here: a comment) or sent with the query. Comments have various features indicating how users interacted with the comment, as well as features computed from the comment content itself. In addition, the system keeps track of the reputation of each comment author as a feature.

User actions are sent as update operations to Vespa as they are performed. The information about authors is also continuously changing, but since each author can write many comments it would be wasteful to have to update each comment every time there is new information about the author. Instead, the author information is stored in a separate document type — one document per author, and a document reference in Vespa is used to import that author feature into each comment. This allows updating the author information once and have it automatically take effect for all comments by that author.

With these features, it’s possible in Vespa to configure a mathematical function as a ranking expression which computes the rank score or each comment to produce a ranked list of the top comments, like the following:

Using a neural net and reinforcement learning

The team used to rank comments with a handwritten ranking expression having hardcoded weighting of the features. This is a good way to get started but obviously not optimal. To improve it they needed to decide on a measurable target and use machine learning to optimize towards it.

The ultimate goal is for users to find the comments interesting. This can not be measured directly, but luckily we can define a good proxy for interest based on signals such as dwell time (the amount of time the users spend on the comments of an article) and user actions (whether users reply to comments, provide upvotes and downvotes, etc). The team knew they wanted user interest to go up on average, but there is no way to know what the correct value of the measure of interest might be for any single given list of comments. Therefore it’s hard to create a training set of interest signals for articles (supervised learning), so reinforcement learning was chosen instead: Let the system make small changes to the live machine-learned model iteratively, observe the effect on the signal used as a proxy for user interest, and use this to converge on a model that increases it.

The model chosen here was a neural net with multiple hidden layers, roughly illustrated as follows:

The advantage of using a neural net compared to a simple function such as linear regression is that it can capture non-linear relationships in the feature data without anyone having to guess which relationship exists and hand-write functions to capture them (feature engineering).

To explore the space of possible rankings, the team implemented a sampling algorithm in a Searcher to perturb the ranking of comments returned from each query. They logged the ranking information and user interest signals such as dwell time to their Hadoop grid where they are joined. This generates a training set each hour which is used to retrain the model using TensorFlow-on-Spark, which produces a new model for the next iteration of the reinforcement learning cycle.

To implement this on Vespa, the team configured the neural net as the ranking function for comments. This was done as a manually written ranking function over tensors in a rank profile. Here is the production configuration used:

rank-profile neuralNet {  function get_model_weights(field) {    expression: if(query(field) == 0, constant(field), query(field))  }  function layer_0() { # returns tensor(hidden[9])    expression: elu(xw_plus_b(nn_input,                              get_model_weights(W_0),                              get_model_weights(b_0),                              x))  }  function layer_1() { # returns tensor(out[9])    expression: elu(xw_plus_b(layer_0,                              get_model_weights(W_1),                              get_model_weights(b_1),                              hidden))  }  # xw_plus_b returns tensor(out[1]), so sum converts to double  function layer_out() {    expression: sum(xw_plus_b(layer_1,                              get_model_weights(W_out),                              get_model_weights(b_out),                              out))  }  first-phase {    expression: freshnessRank  }  second-phase {    expression: layer_out    rerank-count: 2000  } }

More recently Vespa added support for deploying TensorFlow SavedModels directly (as well as similar support for tools saving in the ONNX format), which would also be a good option here since the training happens in TensorFlow.

Neural nets have a pair of weight and bias tensors for each layer, which is what the team wanted the training process to optimize. The simplest way to include the weights and biases in the model is to add them as constant tensorsto the application package. However, with reinforcement learning it is necessary to be able update these tensor parameters frequently. This could be achieved by redeploying the application package frequently, as Vespa allows that to be done without restarts or disruption to ongoing queries. However, it is still a somewhat heavy-weight process, so another approach was chosen: Store the neural net parameters as tensors in a separate document type in Vespa, and create a Searcher component which looks up this document on each incoming query, and adds the parameter tensors to it before it’s passed to the content nodes for evaluation.

Here is the full production code needed to accomplish this serving-time operation:

import com.yahoo.document.Document; import com.yahoo.document.DocumentId; import com.yahoo.document.Field; import com.yahoo.document.datatypes.FieldValue; import com.yahoo.document.datatypes.TensorFieldValue; import com.yahoo.documentapi.DocumentAccess; import com.yahoo.documentapi.SyncParameters; import com.yahoo.documentapi.SyncSession; import com.yahoo.search.Query; import com.yahoo.search.Result; import com.yahoo.search.Searcher; import com.yahoo.search.searchchain.Execution; import com.yahoo.tensor.Tensor; import java.util.Map; public class LoadRankingmodelSearcher extends Searcher {    private static final String VESPA_ID_FORMAT = "id:canvass_search:rankingmodel::%s";    // https://docs.vespa.ai/documentation/ranking.html#using-query-variables:    private static final String FEATURE_FORMAT = "query(%s)";      /** To fetch model documents from Vespa index */    private final SyncSession fetchDocumentSession;    public LoadRankingmodelSearcher() {        this.fetchDocumentSession =           DocumentAccess.createDefault()                         .createSyncSession(new SyncParameters.Builder().build());    }    @Override    public Result search(Query query, Execution execution) {        // Fetch model document from Vespa        String id = String.format(VESPA_ID_FORMAT, query.getRanking().getProfile());        Document modelDoc = fetchDocumentSession.get(new DocumentId(id));        // Add it to the query        if (modelDoc != null) {            modelDoc.iterator().forEachRemaining((Map.Entry<Field, FieldValue> e) ->                addTensorFromDocumentToQuery(e.getKey().getName(), e.getValue(), query)           );        }        return execution.search(query);    }    private static void addTensorFromDocumentToQuery(String field,                                                     FieldValue value,                                                     Query query) {        if (value instanceof TensorFieldValue) {            Tensor tensor = ((TensorFieldValue) value).getTensor().get();            query.getRanking().getFeatures().put(String.format(FEATURE_FORMAT, field),                                                 tensor);        }    } }

The model weight document definition is added to the same content cluster as the comment documents and simply contains attribute fields for each weight and bias tensor of the neural net (where each field below is configured with “indexing: attribute | summary”):

document rankingmodel {    field modelTimestamp type long { … }  field W_0 type tensor(x[9],hidden[9]) { … }  field b_0 type tensor(hidden[9]) { … }  field W_1 type tensor(hidden[9],out[9]) { … }  field b_1 type tensor(out[9]) { … }  field W_out type tensor(out[9]) { … }  field b_out type tensor(out[1]) { … } }

Since updating documents is a lightweight operation it is now possible to make frequent changes to the neural net to implement the reinforcement learning process.

Results

Switching to the neural net model with reinforcement learning has already led to a 20% increase in average dwell time. The average response time when ranking with the neural net increased to about 7 ms since the neural net model is more expensive. The response time stays low because in Vespa the neural net is evaluated on all the content nodes (partitions) in parallel. This avoids the bottleneck of sending the data for each comment to be evaluated over the network and allows increasing parallelization indefinitely by adding more content nodes.

However, evaluating the neural net for all comments for outlier articles which have hundreds of thousands of comments would still be very costly. If you read the rank profile configuration shown above, you’ll have noticed the solution to this: Two-phase ranking was used where the comments are first selected by a cheap rank function (termed freshnessRank) and the highest scoring 2000 documents (per content node) are re-ranked using the neural net. This caps the max CPU spent on evaluating the neural net per query.

Conclusion and future work

In this article I have shown how to implement a real comment serving and ranking system on Vespa. With reinforcement learning gaining popularity, the serving system needs to become a more integrated part of the machine learning stack, and by using Vespa this can be accomplished relatively easily with a standard open source technology.

The team working on this plan to expand on this work by applying it to other domains such as content recommendation, incorporating more features in a larger network, and exploring personalized comment ranking.