If you had asked for seafood joints, we could have flooded you with recommendations, but as to burgers, beyond the usual fast food joints, which you can evaluate for yourself, there are a few places not specializing in burgers which you might consider.
I have eaten at the outlet cafe and do not recommend it.
In the town of Clinton, Aqua restaurant on the shoreline has Bacon-Cheddar, Gorgonzola and Mushroom-Swiss burgers. I've only tried the last one--it was ok. The other place where I once had an ok burger was Chips Pub III on Main St., which has typical bar food.
In nearby Westbrook, Edd's Place on Rt. 1 is a combination deli plus restaurant with a view of a saltwater marsh from his gazebo, and has burgers, dogs and meatloaf as well as other stick-to-the-stomach food. The New Deal Steakhouse, Lenny & Joes Seafood and Bill's Seafood all have burgers (and some dogs) on the menu. The New Deal may be your next best bet. I know Bill's are ok, though, since I had one while playing in a jazz band there.
If you want to go a bit further, Johnny Ad's in Old Saybrook has famous foot-long dogs as well as seafood, and I hear the Killingworth Cafe in the town N. of Clinton has good burgers as well as steaks and ribs. In Madison, only place I can think of for burgers is Friends & Co. on Rt. 1, although they are only on their brunch menu.
Lenny & Joe's
Westbrook, CT, Westbrook, CT