Just a question, can they remember something from the past bits by bits? or they can only remember the 3 things that remained?
Hmm, that's a tough question// I'd say that the things they picked as memories can be improved— but they can't remember everything. Doing things they had previously done before entering the mysterious town may trigger a familiar feeling though, even if you didn't have that as one of your memories.
Basically, the chosen memories can be enhanced, but there will always be things missing.
Plus, there's not really a set-in-stone way that the memories work, just think about it and decide how you want your characters to develop within the group and go from there//