In sajdah, one should recite سُبْحَانَ رَبِّيَ الْأَعْلَى thrice.
Reciting the following tasbeeh in sajdah is also established in the sunnah:
سُبُّوحٌ قُدُّوسٌ رَبُّ الْمَلَائِكَةِ وَالرُّوحِ
You are glorified, You are pure, You are the Rabb of the angels and the Rabb of Jibreel (alayhis salaam)
عن مطرف بن عبد الله بن الشخير، أن عائشة نبأته أن رسول الله صلى الله عليه وسلم كان يقول في ركوعه وسجوده سبوح قدوس رب الملائكة والروح (صحيح مسلم، الرقم: 487)
Hazrat Aaishah (radhiyallahu anha) reports that Rasulullah (sallallahu alaihi wasallam) would recite the following tasbeeh in his ruku and his sajdah:
سُبُّوحٌ قُدُّوسٌ رَبُّ الْمَلَائِكَةِ وَالرُّوحِ