He hailed from Langya 琅琊 (modern Zhucheng 諸城, Shandong) and was famous as a Confucian scholar that well understood the ancient Classics. He was therefore appointed erudite (boshi 博士) at the National University (taixue 太學).
Later on he became regional inspector (cishi 刺史) of the province of Liangzhou 涼州 but soon retired because of illness. After some while he was appointed magistrate (ling 令) of the district of Henan 河南 but after a year again withdrew from office.
When Emperor Yuan 漢元帝 (r. 49-33 BCE) mounted the throne, he was made Grand Master of Remonstrance (jianyi dafu 諫議大夫), together with Wang Ji 王吉. In that position he regularly submitted memorials explaining that the government was not functioning quite well and that corruption and negligence prevailed at the court. He suggested punishing corrupt officials and promoting competent and responsible persons.
The Emperor acknowledged the correctness of his memorials and had reduced taxes to express the benevolence of his government towards the peasant population. Gong Yu was promoted to Grand master of Splendid Happinesss (guanglu dafu 光祿大夫) and later to Censor-in-chief (yushi dafu 御史大夫).