I thought I'd start part 2 of this thread since the original one was getting too long. Just wanted to update.
I'm reading both "No more Mr Nice Guy" and "How to get a date worth keeping." I've read more of the 2nd one and it is very interesting, insightful and helpful.
I've also reflected on different things in the past few days, and observed other social interactions.
I've noticed that when I am not introspective or feeling depressed, I can project the image of someone who isn't unlucky or unconfident with women. During these times I can be more open, playful and at ease, and I can easily compliment women. People observing me then might think I don't have any problem with women, especially if they were to look at all my photos on Facebook (where I constantly seem to be surrounded by women), or when I'm able to get into that mode similar to when I'm doing public speaking or performing.
When I am depressed, feeling ashamed of myself, introspective or have been recently rejected in some way, or if I've recently masturbated/ viewed porn and am feeling drained of energy or guilty, I cannot relate to anyone, let alone women. I feel more closed off, unspontaneous and very dull in my delivery. People would never know I could ever get up in front of others because I seem more shy and reserved.
I've also observed other friends who have had similar problems with women (but not necessarily with finding work). One particular person seemed very broken and embittered, and very closed off. I noticed that even I tended to avoid him slightly, especially when he was complaining of essentially being a failure in life. I can relate to his situation (although he has had relationships but perhaps not many). However, while I may vent my frustrations here on this forum, I never suggest publicly in real life to others what may be going on internally in my mind - I don't tell others that I think I'm a failure or have no hope with women. I realise that if I were to say anything that I've said in my previous thread to people in real life, rather than sympathising, it would probably put them off. That may be even be the case here, but since this forum is specifically designed to deal with these sorts of topics, I guess it's a little more acceptable. It's interesting observing others who are in similar situations and the reactions they get from others or even myself, because then I can use that as a mirror into my own life. I also notice that this friend is generally less confident than me in many areas and certainly less surrounded by women most of the time. Probably, as he sees it, I have no difficulty finding women or finding a relationship, even though that couldn't be further from the truth.
I try to act popular in the hope that by acting that way, it will become a reality. It seems that popularity attracts more popularity, whereas if you are unpopular, even the few friends you do have eventually disappear. I wouldn't say this solves my problem at all, but I guess in comparison to my friend mentioned above, it does help me to have more friendships and opportunities.
Now I just need to be able to put some of these ideas in the books mentioned above into practice, once I've finished reading them.
Thanks for people's suggestions and advice before.