When it was obvious that the internet wasn't a fad, lots of people worried about it opening all sorts of worm cans. And, in some instances, they were right. But yet the benefits have largely outweighed the detrimental aspects. For example, a site that helps strip club hobbyists get the best... errr... bang for their buck.
I don't know of any conceptual version of VR where the user is completely unaware that it's fake. But sure, you could extrapolate VR to highly unethical uses once the technology reaches Star Trek levels. Example: interrogation. If you're waterboarded in VR, then have you actually been tortured?
The other negative result is that certain personalities may become lost, disconnected, or obsessed within VR. That type of dysfunction could become an issue well before "holodeck" VR.
But unhealthy, obsessive fantasy behaviors are not new and already present minus advanced VR. Example: San Jose Guy.
It's probably going to be a bumpy road, but we'll travel it anyway. Humans have essentially never turned away from technology based on ethical concerns.