Sydney Lower North Shore suggestions
Not much in the way of quiet near Milson's Point. It's mainly cramped Thai restaurants. Bungalow Thai is less cramped, though. Best bets might be Ricky's (cafe), Garfish (seafood) or Ainoya (Japanese - grab a hori-gotatsu seat).
Also, the new Italian place (La Capannina) isn't bad, and great views of the harbour if you get the right seat. The Deck has good views too, and is quiet most days of the week (but stay away from Ripples -- terrible). Milson's might prove a bit too quiet for your liking mid-week. Delicado foods (tapas) on Blues Point is nice, but might be a bit of a hike for people coming off public transport (?)
I like Crow's Nest. Sakana-ya is probably my favourite Japanese place around (their simmered fish is great). I also like Mihana (for their simmered dishes like oden and chikuzen-ni). Ju-Rin is great, but may be too busy for your liking. If you don't get a bus to Crow's Nest, you can walk from St Leonards.