I'm 6'4"
Get like a 10-20lb dumbbell and do shoulder lateral raises then put like 20-30lbs of stuff in a backpack and do pushups if you're having trouble.
Tricep, Back and shoulder muscles are the main thing.
What I do (Did before I sold my bike, I also lift weights now.) Is all mental, I always mind my riding form, I rode with improper form for about a month then I learned about "proper" riding form and from there forced myself to do it right.
It's like riding boots, I used to HATE how stiff they were, Now I refuse to ride without them, The support and protection gives me a huge confidence boost.
But again, I'm a racer, I'm all about doing it "right" not comfortable, I guess I find comfort in doing it right.