It takes three hours to travel to Peng's residence, which he shares with seven other fishing families, on a small and isolated island in the center of the lake.
They live in an old reed-processing factory, but will have to move out in the reed-harvesting season, when the factory will start operating again.
"We are living like barbarians and life is getting harder," says Peng.
"Although we work hard, my family still owes more than 50,000 yuan ($7,885)," says his 26-year-old neighbor Zhou Bing, who often fishes from first light and well into the evening.
Zhou, who has only a primary school education, says he and his father earn about 30,000 yuan ($4,700) a year, but nets and other annual costs run to more than 10,000 yuan.
During the eight months of the close season for fishing, Peng and his neighbors harvest and sell wild vegetables, or work at the reed factory.
He Daming, however, has sold his fishing nets and opted to borrow money and start a small restaurant on the lakeside.
"I've seen a dramatic drop in the kinds and number of fish in the lake in two decades," says the 43-year-old, explaining his decision.
"If the situation continues, I fear there will be no more fishermen in the next generation."
Tan Yiqi adds that water loss due to reclamation and upstream water projects and soil loss have affected fish reproduction, and recent droughts have also damaged the fish population.
He says this, compounded by illegal fishing and insufficient manpower and resources, makes it difficult for the authorities to rectify the situation.
In response, the Yueyang authorities are urging fishermen to switch careers and have rolled out incentives to do so.
Since 2009, this has included relocating fishermen and providing accommodation, retraining, plus free education for children.
Some, like He, have quit fishing, but others are reluctant to do so.
While Peng says he is willing to give up fishing, he is not reassured about his future. "Simply put, having a house is not enough because we must make a living."
"I have thought about trying something else," says Zhou, but adds he doesn't have the necessary skills. He says he tried to find a job in the city five years ago but was duped into joining a pyramid marketing organization.
The fishermen argue that the retraining provided by the local fisheries department, such as driving and computer skills, is not helpful.
"We cannot even write our names and know nothing about computers. Even if we learn to drive, there is no promise of a job," Peng says.
He suggests the local government instead finds jobs like working in a factory.
As for pollution, less water has reduced the lake's ability to cleanse itself, leading to signs of eutrophication in some parts, comments a recent report by a research team headed by Lai Sheguang, chairman of the city's political consultative conference.
Though Yueyang has closed 64 polluting factories from 2005 to 2010 - mainly in the papermaking, textile printing and dying industries - much work needs to be done to improve the lake's ecology.
The report adds that because the lake is administered by different cities or counties, it is difficult to tackle problems, such as ensuring fishing is done in a legal and coordinated basis.
"We believe that all the regions around the lake, including its upstream areas, must work together now to improve the lake's condition," says Li Guobao, chief of Yueyang's environmental protection bureau.
Li agrees with a CPPCC member's recent proposal to establish an integrated environmental-economic circle around the lake, which he believes could solve the longstanding problems.
Contact the writer at liuxiangrui@chinadaily.com.cn.