Neoprene is a synthetic rubber, so I don't know how that would work. I would place something between it and your filter though since it's closed cell synthetic rubber. When you look at the white, cut-foam material that is normally used for filter protection in those round, clear filter cases, it's usually of a different type of material. Any open celled foam sheet would work.
Logically (for what that's worth), if debris gets between your lens and whatever filter protection material you use, it may just go into the open cell part rather than be mashed against the filter glass.