Start with the scored skin down in a cold pan and turn it to med-hi. Don't add extra fat. The skin will crisp up in its own fat, and there shouldn't be any fatty layer left, save for perhaps a tiny one.
I cook duck quite often and I never really get rid of the fat layer completely - a lot of it sure, but completely no. I like that fat and would definitely include the skin and any small layer of fat that remains on each slice.
I made seared duck breast for the first time this week. The skin was great but I admit the meat itself was pretty tasty. Does anyone remove the skin and just enjoy the meat or is the skin a primary reason that you're eating it?