Doing business at a known place of criminal, organized activity, and returning, could be seen (I strongly emphasize "could") as complicity, even facilitation. His cash to you on that site could potentially be a form of money laundering. I would advise not to return. I am speaking from some experience here having had to deal with the fallout from grow-ops in Vancouver relating to real estate and utility transactions, watching willfully ignorant tradespeople given the nth degree.
The local cop is not always going to listen to you or read your contract if your name comes up as having been to that site. He's going to follow the money and the personnel at that site.
You are putting yourself in a very bad position by being in the presence of whatever unlawful activity this client is involved in.
If the police show up when you are there, you could easily be arrested as an accomplice and have all your equipment confiscated as evidence - and then have to post bond to get out of jail and spend thousands - even tens of thousands - on attorneys to defend you at trial and hopefully keep you out of prison.
Forget public defenders; they are overloaded with cases and do not have the time and resources to mount an effective defense. Public defenders are an excellent way to end up doing prison time or being forced to plead guilty to a crime you had no involvement with.
If you continue to associate with this client and do business with him, do NOT do it at his home.
Tell him you will need to rent a studio and bill him for it. If he objects, explain to him why as tactfully as you can. If he still objects, say "Thanks but no thanks" and cease all contact with him.
No job is worth risking what you are risking, no matter how much it pays.
JMHO.
EDIT: Just found the OP's last entry on this situation. I'm glad to hear you have extricated yourself from this guy and his "activities." Good call.