Two points:
- "Dangerous" and "toxic" are relative terms. Something can be more dangerous than another thing but still safe enough for whatever one's comfort level is.
- Even B&W processing varies in danger. A pyro developer is, by most accounts, more dangerous than XTOL. Ever hear of uranium toner? It was somewhat popular once.
I don't claim to be an expert on it, but my understanding is that E-6 (and other color processes) involve chemicals that are nastier than the average B&W chemical, but these chemicals aren't so nasty that they're unreasonable to use in a home darkroom. Read the documentation that comes with whatever kit you buy (and/or check up on the manufacturer's Web site) and follow its advice and you should be fine. I certainly wouldn't describe E-6 processing as "quite dangerous," although as I say in point #1, "dangerous" is a relative term, so that's a matter of what your reference is.