Why AI “Forgets” Your Conversation

Key Takeaways

Have you ever noticed that after a long chat with an AI tool like ChBT, the LLM suddenly stops remembering some things you've said in the past? That's because memory doesn't exist in their case. What we actually do is that we try to fill up the space that they have access to, which is the context length to the maximum possible. In most cases, it's up to a few thousand tokens. And so if your chat story is very long, we sometimes have to build around that as an AI engineer and do some kind of tricks to make the whole discussion fit into the model's context. Which means that we use tricks like summarization or embeddings to compress the previous messages and send everything possible or send everything relevant through building retrieval systems. In any case, the model doesn't have access to the whole history, especially if the discussion is super long. And so, it might be best to start a new conversation when you change topics or when you want to improve results because its past responses have been a bit weird or forgetting some things you've mentioned. And if you have a very long discussions with this system and ask, "What have we been talking about?" It will use some retrieval and find what we call the closest embeddings to then retrieve all the previous useful messages to answer the current query. It's not a real memory as we have. It's something we create as engineers for DLM to have the model keep the conversation flow. I'm Louis Frano, CTO and co-founder of toi and follow me if you want to learn more about AI.