I've done a lot of "reversal RA-4" prints from color slides. In this process, the first step is a BW developer and then I expose the paper to light, so this is actually exactly what you are asking, just that after the exposure a color developer is used, but I always see the BW negative there before doing that.
So yes, BW developer will work on RA-4 paper and will produce a nice B&W negative. But the Dmax is not very high compared to BW papers, so expect lower contrast. As an extra benefit, you can create a color positive image afterwards if you want.