Projected Autoregression: Autoregressive Language Generation in Continuous State Space

1. 1Replace traditional token selection with continuous prediction in embedding space
2. 2Predict next-token vectors via autoregressive models
3. 3Project predicted vectors to discrete tokens at commitment time
4. 4Evaluate the generated text using standard language evaluation metrics

NLP researchers and AI engineers on a team can benefit from this approach as it offers a new perspective on autoregressive language generation, allowing for more flexible and continuous modeling of language

💡 Autoregressive language models can be designed to predict continuous vectors in embedding space instead of discrete tokens, allowing for more flexible modeling of language

💡 Projected Autoregression: a new approach to autoregressive language generation in continuous state space