It does but they play in the small town of Nailsworth which is not far from where I lived for many years. I was on the south side of Gloucester and it was only a few miles down the Stroud road and onto Nailsworth but I never visited their ground. I often watched Gloucester City and Cheltenham Town though, who were both a fair way down the non-league pyramid at the time.
The Crystal Palace was the huge glass house erected in Hyde Park to host the Great Exhibition of 1851. Subsequently moved to South East London, it burnt down and the local football club took up the name. Not an area at all, that would be Penge.
It is a place, used to be called Sydenham Hill. It was renamed after the Exhibition Centre. The club then took their name from it.
It's just a few miles down the road from here and not a place I'd visit unless I had to. Quite a lot of South London has evolved and gentrified over the last 20 years but Penge hasn't followed the pace as yet. There are quite a few wooded areas though which were part of an old ancient forest that covered just about all of south east London back in the day. Sydenham Woods are on the way and quite pleasant. Quite close to London Beer factory if I cast my mind back 18 months and prior.
When I was a southerner I visited Selhurst Park a few times and always got off the train at Thornton Heath. Walking distance to the ground.
You're right, it is an electoral ward and there is crystal palace park... home of the infamous dinosaurs! And if you look on google maps the area is considered Crystal Palace.