Without heavy regular use, muscle fibers die (never to be recovered) or atrophy (can be recovered with effort). Such is the fate of the middle-aged body. Young people can add new muscle fibers with exertion, old people can not.
In later years, it is easier to improve strength and endurance than bigger muscles. See discussion.