What is going to be in the upper part of the center structure?

It seems really high unless you are planning on having a third floor. It looks almost 3 times as tall as it is wide through the center. If theres a second floor in there that would help, but if you have any open to above areas from the first floor they might tend to resemble and elevator shaft.

You could keep your peak that high is thats the look you want and lower your control points (change your roof pitch) That would help lower the exterior walls and solve some of the structural problems already mentioned.