Abū ‘Abbās Ahmad ibn ‘Umar al Qurtubī al Mālikī (d.578h) رحمه الله stated whilst explaining the hadith, “Do not prayer towards the graves and do not sit on them,”

“Meaning do not take them as the direction of prayer. This is like what we previously mentioned concerning the prohibition of taking graves as places of prayer and the disparagement of the Jews due to what they did by taking graves as places of worship.

All of this is to curtail all means which may lead to ignorant people believing that by praying toward graves or on them that the prayer is actually to them which leads to worshipping those in the graves similar to the rationale which became the reason idols were worshipped [1].”

[al Mufhim li ma Ashkala min Talkhīs Kitāb Muslim , v.2, pg.628]

[1] This is a reference to the hadith of Ibn ‘Abbās in his tafsīr of { And they said: ‘You shall not leave your gods: nor shall you leave Wadd, nor Suwâ‘, nor Yaghûth, nor Ya‘ûq nor Nasr’ (these are the names of their idols) [ Nūh:23] with regards to the people of Nūh [al Bukhārī, no. 4920] and how they fell into shirk by erecting statues of their righteous figures to remind them to be upright. Yet, when that generation died and the people became ignorant, they began to worship those idols.