Great upscale dinner in Portland, ME?
The two things that need to be quantified are the words "fancy" and "family". The usual suspects of Street & Company and Fore Street are certainly "fancy" by most standards, with 555, the Grill Room and Old Port Sea Grill slightly behind. That being said, none of those are places where you'd either ask for or expect to find a kids menu.
I agree with HDinCentraME - Portland has some excellent food but not so much a fancy scene. Back Bay Grill does provide the most refined atmosphere. Hugo's has/had a fancier atmosphere, but research it -- it doesn't work for everyone. Cinque Terre perhaps, has a bit of a fancier vibe. 555 provides white table-cloth fine-dining level service and good food. The atmosphere is not particularly fancy, although you *may* find a more a more intimate experience on the upper level. Evangeline would be along those same lines - excellent food and service, but more casual. The Grill Room is very good and fun, but LOUD and certainly festive, for steak.
I'll stop now. The "excellent food" criteria is pretty easy to meet, but fancy -- not so much... unless you go by relative price tag :)
There are quite a few restaurants that have excellent food but most/all of these are not "fancy". Open kitchens/relaxed atmospheres are the norm.
Consider the Back Bay Grill as on the "refined" end of the scale but not stuffy.
Back Bay Grill
65 Portland St, Portland, ME 04101