You sound like by brother Alpine. He has almost no body fat and has worked out since his mid-teens (he's 43 now). I lived with him 15 years ago when I was going through college and when he got the flue a couple times, he'd lose 20lbs in a week. His metabolism was/is so high that if he isn't working out and eating like a mule, his body eats itself up almost.
I used to work out like a fiend in high school, grades 7-12. From grades 10-12, 3 hours a day m-f year round. Was young and stupid. Always was a bit heavy and never got "cut" like a few of the guys I worked out with. From 11-12, there was only one guy I could work out with that was lifting the same weight, so was easier/quicker to not have to change weights all the time. He weighed nearly 350lbs, 6'4", I was 225lbs 5',11", in my prime.
Then he and I met up on the football field at practice right before our homecoming game, doing a stupid drill that the team had done for decades. Head on, full tilt, one on one. Cracked two vertebrae in his spine, dislocated his shoulder and destroyed my right shoulder.
Was never really able to work out again, not without severe pain. Once I got to a certain point, then that was taken from me with a promise of never being able to lift like that again... pretty much makes going to the gym just a constant reminder of what I can't be.
I tried working out a couple years ago... didn't go well, could barely bench my weight without searing pain. Although, they had a leg press I never had in high school. Ours was an annoying pivot type machine that had foot rests that dug into your feet when doing a lot of weight.
