Loretta Lynn Explains What Fame Cost Her And What She Can Never Get Back 

Loretta Lynn cut her first record, “I’m a Honky Tonk Girl,” in 1960, she split her time cleaning houses and taking care of her four young kids. But soon after her clear Kentucky voice hit country radio with songs of strong — and often wronged — women, the 28-year-old singer songwriter found herself on show circuit away from family most weeks.

“It was rough,” Lynn, now 84, tells PEOPLE of her early days on the road. “I was working clubs, and I was doing like three and four shows a night. And I hadn’t been in the business but two years before I got pregnant, and I had twins. I’d play shows right up until the babies was born and that guitar around my neck just about killed me. I don’t advise it to any mother.”

Most years, Christmas meant a welcome break from her touring schedule. “I would try to make it home for the holidays, but sometimes I was late,” she says, recalling the year her bus got stuck in Texas and her kids had to wake up without her on Christmas morning. “Just close your eyes and think about your own family – just think about being away from them on the holidays. It was hard for me. Sometimes I’d cry.”

But, Lynn says, she felt she didn’t have a choice. She and her husband, Doo, a mechanic with a meager salary, had moved the family from their home in Washington state to Nashville betting on her success. And Doo, who was both encouraging and controlling of her musical career, pushed her toward it.

“I knew if I was going to make it, I had to be on the road,” says Lynn. “I knew my husband was going to make me sing, so I had to do it. I had to sing.”

For more on Loretta Lynn and to see how she celebrates the holidays with her big Tennessee family, pick up the latest issue of PEOPLE, on newsstands now.

Lynn eventually became the biggest thing in country music, winning the Country Music Association’s first female vocalist award in 1967 and becoming the first woman to take home the CMA’s Entertainer of the Year prize in 1972. She’s sold more than 45 million albums worldwide with songs like “Fist City,” “Coal Miner’s Daughter,” and “The Pill” but, she says, looking back on the time away from family is still hard: “You never catch up the lost time. That time’s gone.”

These days Lynn’s expansive family (four surviving children, 20 grandchildren, 24 great-grandchildren and three great-greats!) come to her on the holidays. The Lynn clan gathers at her Hurricane Mills, Tennessee, home on Christmas Eve for prayer, presents — and Memaw’s famous chicken and dumplings, so good even Lynn’s pal (and her 2004

album producer) Jack White has raved about them.

And though the singer hasn’t slowed down — this year alone, she’s released two new albums,

— she does appreciate the time celebrating with her kin, even when it turns into a raucous event. “Family means everything to me,” Lynn says. But with all the little Lynns around, she jokes, “I just put stoppers in my ears!”

1 How Blake Shelton Stopped RaeLynn from Giving Up: ‘I Think the World of Him’

2 Ring in the Christmas Spirit with Festive New Albums from Brett Eldredge, Kacey Musgraves and More!

3 Blake Shelton on His ‘Hot’ Girlfriend Gwen Stefani: ‘It’s Been an Eye Opener to Be with Someone Like Her’

4 Luke Bryan Takes a Swing at Heckler After Being Taunted During Concert

5 Eli Young Band’s Jon Jones Welcomes Daughter Lila Pascale