Your basic physical architecture is a product of a genetic potential, a genotype. Your genetic potential (design) is effected to some extent by environmental influences: intrauterine nutrition, nutrition during growth, disease, physical activity during growth, and so forth. All the same, your genetic potential can not be exceeded because it is pre-programmed.
What do I mean by basic physical architecture? I mean things like basic body structure: height, bone lengths and widths, and number of muscle cells and their distribution (yes, you are born with all of the muscle cells you will ever have, and these vary a good deal between people).
As an adult over age 17- 21, your life-long basic physical foundation is complete. As with your IQ, personality strengths and weaknesses, athletic talents, what we do with our given physical architecture after that point is up to us. A good trainer, a physical therapist, or a sports physician can see through our appearance and phenotype and get an idea of our basic physical architecture and our potential for development of what we are.
Our naked physical appearance today may or may not tell us much about our basic architecture, because it might be hidden under layers of fat or simply unmanifested by lack of stressors and demands. In underweight people, it can be concealed by underdevelopment of potential.
It's common to talk about endomorphs, mesomorphs, and ectomorphs, but that is simplistic because everybody is built differently. The length of your femur, for just one important example.
Do these things matter in general fitness efforts? Not too much. Do they matter in physical training? Yes, because the architecture sets limits on, and offers opportunities for, what can be achieved. Do they matter in athletic potential? Certainly, but it depends on the level which is aspired to.