how does the sentiment model work (for frank)? i did some work on a sentiment analysis thing a while back and it was pretty disappointing imo. like, good for product reviews and not much else. but frank seems to be able to 'have' different moods
Good question. I’m using an off-the-shelf “sentiment analysis” model, specifically the Roberta one shown here. (In that demo, you have to select it in the dropdown instead of “GloVe-LSTM.”)
So, yeah, it’s trained on movie reviews (Stanford Sentiment Treebank) and has all the limitations you’d expect from that. Although, as a BERT model, it probably generalizes better than older models since it’s leveraging so much prior knowledge from pre-training.
I originally got into doing sentiment analysis with this bot as part of the reblog-from-dash feature, when I wanted a more accurate screener to prevent Frank from reblogging really posts with really sad/heavy content. Then I had the capability, and said, hey might as well use it for more fun stuff. I don’t expect very much out of it, and it’s done . . . decently? Maybe better than expected?
I use it in 3 ways in nostalgebraist-autoresponder:
When trying not to reblog sad posts from dash, I just run the sentiment model on the posts, and I have a cutoff on the output.
When generating new posts that fit a given “mood,” I run the sentiment model on each candidate post, and reject anything outside upper and lower bounds given by the “mood.”
(I did a bunch of tuning to get reasonable bounds that move up and down with a scalar “mood” variable, some more about this here)
When determining how Frank’s mood should be affected by an ask/reply/etc. . . . actually this one has changed.
Originally, I just got the sentiment of the ask/reply/etc., as with the sad-post screener. However, this failed in cases where a brief text looked different out of context than in context (e.g. “that sucks” gets a very negative score, but is a positive gesture in context).
What works better -- I did some annotations to establish this -- was checking the sentiment of all generated responses (incl. the ones we’ll eventually reject from the current mood bounds), and using a summary stat over those to determine the impact of the input on near-future mood.
You can think of this like, “if a conversational text generator mostly produces happy responses to an input, then that input is the kind of thing that makes a person happy when it is said to them,” and likewise with “happy” replaced by “sad”
This is getting further from the topic of your question, but for completeness and since I had a draft written about it:
The “mood value” itself -- the thing which responds to user input and determines bounds for output -- is the sum of a daily-baseline component that changes every 24h, and a dynamic component responding to user input.
The dynamic component is a 2nd-order LTI system. It looks like
d(mood_dynamic)/dt = -mood_dynamic/tau_0 + hidden
d(hidden)/dt = -hidden/tau_1 + user_input
where tau_0, tau_1 are time constants, and user_input is treated like a delta spike (any user input event instantaneously kicks “hidden” up/down, i.e. kicks the derivative of “mood” up/down). I could talk more sometime about how I picked this, but as with most things autoresponder, it’s the simplest thing that felt reasonable.
Also -- technically, what you see in the mood graphs is the underlying mood variable mapped into [0, 1] with 1/(1+exp(-x)).
This is the probability space of the sentiment model. For most computations using sentiment model output, I feed probabilities through the inverse of that function (equivalent to using the difference between model logits) and work in this “logit difference” space. Like many modern neural net models, this one tends to spit out probabilities very close to 0 or 1, so the metric of the “logit difference” space is more well-behaved: in probability space all differences look very small except the big difference between “close to 0″ and “close to 1.”
