If your gluing something to wood, like felt, I would use Elmers Wood Glue, or even plain old Elmers glue. I use it a lot for wood to wood projects. The felt would absorb so much of the glue the stuff will probably never come off, what is what I think you would want. Just make sure if the wood has been painted the area to be glued is sanded so it is a little rough, so the glue has something to stick too. The best way would be not to paint the area at all. There are also some other wood glues similar to Elmers Wood Glue, but is basically the same stuff, looks like kinda light tan color, if you cant find Elmers. It shouldn't take too much glue, just apply it evenly.
James,