I absolutely love ibiza town and can't understand how so many people just ignore it.
I wouldn't go in the middle of the day in july and august - it's simply too hot often. but late afternoon, towards the evening is a great time. walk up and explore dalt vila before sunset, then have a nice dinner somewhere (there are lots of great places), do a bit of shopping if you like (a lot of the shops are open until midnight in summer) and then go for a few drinks in the port - again huge amount of bars to choose from.
honestly, if I still was a holidaymaker on ibiza, I'd spend a lot of evenings in town instead of any of the big holiday resorts.