Welcome to the forums.
It should just pull off. It don't have to be lined up any certain way. Try tapping on the end of #4 with a block of wood and a hammer and try to push it through as you pull the face plate off. Be careful not to damage the end of #4 or you won't be able to get the brake lever back on. If you can push #4 through and pull the face plate off, you should be able to see what's hanging it.It sticks and wont come off. Any ideas? Does it have to be lined up on the shaft in a certain way?