Memories Are What Matters