The Exorcist reminds me of two other movies, wait, three other movies that scared the crap out of me w/o even seeing them. One is too good for my criteria, that is Alien, which I had nightmares about years before I actually saw it, though I did read the novelization by Alan Dean Foster.
But the number two movie that does fit my list, somewhat, that scared me and gave me nightmares w/o ever seeing it was the Larry Cohen classic It's Alive:

Sadly, when I finally did see the movie, it was too cheesy and didn't live up to the hype my mind created.
The other movie that gave me nightmares was Humanoids from the Deep. I thought for sure these bad boys were sneaking into the swimming pool over night!

The thing that made it so scary was they were humanoids, from the deep, which really sent my imagination into overdrive. I had no idea what they could be. The actual movie also doesn't live up to the hype in my head but I think it is a pretty good monster movie overall. At least that is what I thought last time I saw it.