Don't know about brunch, but lunch at any of them should be good. If you are staying at the Hilton downtown, you are an easy walk to the Mayflower (less than a block)... They should all be hung over pretty badly at Babalu because of Babaloosa the night before.