The pin is being held in by corrosion! Once you have banged it out, and probably peened the end over a bit, file the sharp edge off then get a wire brush that fits through the hole in the peg and the holder (bizarrely a 7.62mm weapon cleaning brush in a drill is superb) and clean it all out. When you put it back grease it.
I'll see if I can find a picture of the brush I use (from my army days )