So, Tom Brady is finally retiring. The longtime NFL quarterback has, of course, made such announcements before. Maybe this time it’s for real? We’ll see.

It’s understandable that it’s so hard to say goodbye. Plenty of people with less-glamorous jobs have trouble retiring. And about half of Americans actually don’t retire voluntarily; research led by Boston University’s Alicia Munnell has found that 1 in 2 of us gets forcibly retired, whether because of a health crisis, the need to step back to care for an aging spouse or when we get laid off and can’t find a new job.

In our younger years, many of us look forward to the day when we’ll be able to step away from the rise-and-grind effort of full-time work. Then, as that day nears, we push it further and further into the future.

Some 42% of Americans over age 50 say they will keep working after they reach retirement age. Some of this is surely due to financial need — something Brady likely doesn’t need to consider — but some of it is because, for all we complain about it, we like work. It gives us a sense of identity, a sense of purpose, a set of comrades, a schedule. These are all things we miss in retirement.

Although we spend our whole careers anticipating the time when we can shed workaday obligations, some of us start to feel very unsettled when the moment arrives.

Life transitions are always hard, perhaps none more so than those that go to the core of our identity. And especially in the U.S., it’s common to strongly identify with your work. Even the English language is complicit: We don’t say “I write” or “I teach” or “I practice law.” We say, “I’m a writer,” “I’m a teacher” or “I’m a lawyer.” I can only imagine that separating your work from your self is tougher when you’re a seven-time Super Bowl-winning quarterback.

To perform at the highest levels takes a single-minded focus. It often means spending long hours training, away from your family. It robs you of time with your kids that you’ll never get back.

For an elite athlete, the robbery starts even earlier — many don’t get to experience their own childhoods, much less their children’s. Last week, The New York Times profiled golfer Anthony Kim, once hailed as the next Tiger Woods before disappearing from the sport before the age of 30. Why would such a prodigious talent quit so young? Perhaps because becoming so good, so young came at such a high cost.

When we make such sacrifices for our jobs, perversely it can also lead us to become even more committed to them. A study of surgeons, for example, found that some would boast of the number of divorces they had been through, as if the level of their personal suffering correlated directly with their prowess in the operating room. When you have sacrificed three marriages for your job, it can make it even tougher to get any distance from it — any sense that you are a person who exists outside of work. After all, if it has taken so much from you, it has to have been worth it. Right?

Right?

I think on some level, we can all understand this even if our own careers couldn’t be more different from that of a sports superstar. When Serena Williams announced her departure from tennis last year, her wrenching explanation resonated so deeply that it left tears running down my face. I have no idea what it feels like to win 23 Grand Slams, or even successfully return a single tennis serve. But as a working mom, I do know what it is to rail against the unfairness of a universe in which I can only do one thing, in one place, at a time.

Of course, when super-talented people retire before they are completely spent, as observers we can feel slightly cheated. When someone is as talented as Kim, Williams or Brady, using that talent to the utmost can almost seem like an obligation. But to expect that of anyone, even of ourselves, if we’re lucky enough to be even modestly talented, is to essentially be hijacked by our own talent. The talent is in the driver’s seat and we’re just coming along for the ride.

Stepping back — whether to retire, to change careers, to spend more time with family — is, perhaps, a way to grab the steering wheel back.

Sarah Green Carmichael is a Bloomberg Opinion editor. Previously, she was managing editor of ideas and commentary at Barron’s and an executive editor at Harvard Business Review, where she hosted “HBR IdeaCast.” This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.