The Swizzle Inn is fun and it's right near the ice cream place (link below) we went to last year.
The Spot off of Front Street has excellent Bermuda fish chowder.
The Lobster Pot is a good casual place, though all places in Bermuda are a bit on the expensive side compared to the states.
In addition to a rum swizzle, be sure to have a Dark & Stormy (Gosling Bermuda black rum and Bermuda ginger beer).
Below is a short thread from Feb. '06
With a kid I would search out rum rasin or black walnut ice cream. I remember as a youth having the fish and chips at the White Hourse as a fun time.
A bit off topic but related:
The public stocks and possibly a dunking chair are in the square at St. Georges. Good photo op w/ a child.
The caverns, and the associated perfume factory, might be an interesting this to do.
Go take a picture of the Bobby directing traffic on Front Street in Hamilton.
There is a light house on the south shore that you can go up in.
The navy museum at the Navy Yard is a possible outing.
The aquarium in Flatts is also good with kids.
The smell of fennel, the sound of peepers at night, geckos doing push ups....Bermuda...