The temple is associated with the visit of the saint Ramanujacharya to Odisha.[1] Chaitanya Mahaprabhu during his stay in Puri used to see the deity form of Jagannath daily. During Anavasara when Jagannath and his sibling deities were taken to the secret chamber for 15 days, he was unable to see the Lord. So as per legend, Jagannath directed him to go to Brahmagiri and visit the Alarnath temple. Still today the Shila over which Chaitanya Mahaprabhu used to do Sankirtana is there. Many historians opine that Alvars once visited this place, but there is no mention of it in Divyaprabandam. But still the temple is considered one among the 108 Abhimana Kshethram of Vaishnavate tradition.