Shihon-ji was the uji-dera (氏寺, "family temple") of the Kakinomoto clan.[1] It is supposedly the final resting place of the remains of the poet Kakinomoto no Hitomaro and so became a focal point for veneration of him once he began to be worshiped as a god of poetry.[1] A stone tablet commemorating Hitomaro at this spot is mentioned in the Fujiwara no Kiyosuke Kashū (藤原清輔家集),[1] which was likely compiled in the late 12th century.[2]
It moved at some point, the date of which is uncertain, but it was likely established at its present location within the Muromachi period.[1] Until the Edo period it flourished as a place of learning for Buddhist monks practicing waka composition and tea ceremony.[1] In 1732 (Kyōhō 17), the monks of the temple and the physician Morimoto Munenori (森本宗範) erected a stone tablet (歌塚 utazuka) in the temple grounds.[1]