FYI - for your next trip there is a small Farmer's Market in Sag Harbor on Bay Street every Saturday June - October 9am - 12 pm. Other than that it's mostly Farm Stands that oped around mid June
There's nothing local except asparagus right now. Easthampton has a great farmer's market next to Nick and Toni's which I know is open Weds in season, not sure about weekends.