Biber Salcasi, the Turkish pepper (paprika) paste is relatively easy to find. It comes in hot and mild and is essentially just pureed peppers, with the consistency of tomato paste. There's a Turkish shop across from the Alston Super88 that sells it, and it's available at the Armenian markets in Watertown. Harissa and Ajvar are also widely available, but each more of a concoction rather than straight pepper paste.
Peruvian pastes (rocoto etc.) are tasty but I'm at a loss now that the HiLo closed. You might also try to search Portuguese versions (massa de pimentao), perhaps in East Cambridge? They usually have a short ingredient list: garlic, olive oil, salt, eg.
I had some of the same thoughts in particular Watertown. To be more specific massa de pimentao is usually available at Courthouse Seafood Market (Cambridge) and Seabra (Somerville). Frio Rico in East Boston is the best bet for Peruvian products, but if Somerville is closer the pastes can usually be found at La Sultana Market on Broadway and if they don't have it you can also try Tony's Foodland & Amigo's market on the lower section of that same road. If you go to La Sultana, you can also check the Broadway Portuguese & American (Fish) Market for massa de pimentao but Courthouse/Seabra turnover is better (and the nearby Super Stop & Shop may also have it but probably for more -- the Malden S&S is great for Peruvian, but expensive.)
360R Bennington St, Boston, MA 02128