'Abdullah b. Zubair (Allah be pleased with him) reported on the authority of his mother's sister ('A'isha) saying that Allah's Messenger (ﷺ) said:'A'isha, if your people had not been recently polytheists (and new converts to Islam), I would have demolished the Ka'ba, and would have brought it to the level of the ground and would have constructed two doors, one facing the east and the other one to the west, and would have added to it six cubits of area from Hijr, for the Quraish had reduced it when they rebuilt it.
وَحَدَّثَنِي مُحَمَّدُ بْنُ حَاتِمٍ، حَدَّثَنِي ابْنُ مَهْدِيٍّ، حَدَّثَنَا سَلِيمُ بْنُ حَيَّانَ، عَنْ سَعِيدٍ، - يَعْنِي ابْنَ مِينَاءَ - قَالَ سَمِعْتُ عَبْدَ اللَّهِ بْنَ الزُّبَيْرِ، يَقُولُ حَدَّثَتْنِي خَالَتِي، - يَعْنِي عَائِشَةَ - قَالَتْ قَالَ رَسُولُ اللَّهِ صلى الله عليه وسلم " يَا عَائِشَةُ لَوْلاَ أَنَّ قَوْمَكِ حَدِيثُو عَهْدٍ بِشِرْكٍ لَهَدَمْتُ الْكَعْبَةَ فَأَلْزَقْتُهَا بِالأَرْضِ وَجَعَلْتُ لَهَا بَابَيْنِ بَابًا شَرْقِيًّا وَبَابًا غَرْبِيًّا وَزِدْتُ فِيهَا سِتَّةَ أَذْرُعٍ مِنَ الْحِجْرِ فَإِنَّ قُرَيْشًا اقْتَصَرَتْهَا حَيْثُ بَنَتِ الْكَعْبَةَ " .
Muslim
The Book of Pilgrimage • Hadith 448