YES,YES, YES, .......Ok...unless this horse has impectable breeding and you can afford to keep him with a trainer/breeder to breed him for you. Fine, but pay to have him with a breeder or trainer. Just because he has that bloodline doesn't mean a tinkers dam.
But there is way too many unwanted horses out there. So geld him.

Many boarding barns are not equipt to deal with stallions, nor want to deal with them. They are a huge liability to their boarders and their horses.
1. He'll be safer for everyone to be around. You say he is aggressive around other horses. Well this is your first clue that he isn't happy and will be always aggressive around other horses. Also he's not broke, and I would guess has very little manners, gets into your space pushes you around. This is not cute nor should you think its OK for him to be this way.
2. For the most part, stallions are not pets. Stallions can be dangerous. Just think about the worst case senario...he gets out, jumps the fence, aggressively goes after some other horse. Not a pretty picture.
3. Stallions make for good geldings. Geldings are happier, they can be turned out with other horses. For the most part, they are safer, happier and you'll have way more fun not having to be on alert for mare's in heat or some little kid running up their pony up into his space. Can you say KICKING, BITING OR OTHER MAYHEM. Not to mention possible legal issues that would insue if your stallion should injury or kill someone.
4. It takes a professional trainer or breeder to handle stallions. They have the facilities and experience to deal with stallions. Just how experience are you with horses in general or stallions? You need to ask yourself some serious questions.
But I say you should geld him, now, not later. Listen to the people that are telling you need to geld him.
5. Good luck.