Transformer Decoder from Scratch

Made a transformer Decoder from scratch using pytorch For autograd and GPU efficiency. Implemented- •Rope positional embedding •Weight tying •Masked multi head attention •KV cache •Custom AdamW •Cosine decay and warm up Trained on wiki text 103 for 75k ish total steps Reached a loss of about 3.5 I'd appreciate some feedback submitted by <a href="https://www.