I’ve used ASP.NET Authentication on a handful of projects. I am using it now as well and I ran into some strange issues that I wanted to pass along. First of all this is one of the most useful post I found on using the WCF Authentication Service. Read that post for a good step by step guide.
I had everything setup and it appeared to be working. When I would call the Login method on the service I got back true telling me everything was working. However, the very next line I would try to call a method on my custom service that resided on that site and I would get an unauthorized request. I pulled my hair out on this for a long time until I found this new .NET Framework 4.0 entry in the web.config.
<machineKey validationKey="Your Key" decryptionKey="Your Decrypt Key" validation="SHA1"/>
This new entry is needed for applications that run on separate servers to talk the same language when doing authentication. Obviously you will need generate your own keys for your sites but this fixed it for me. This entry is needed on both the server calling the Authentication Service as well as the server hosting the service.
There could be multiple reasons authentication fails, but this is the one that bit me and took a long time to figure out. Hopefully this will save you the trouble I had to go through to get this fixed.