J.K. Rowling was recently ousted as the author behind a crime novel, The Cuckoo's Calling. According to the article I linked, the book had sold about 1,500 copies before the news broke. Now it's sold a heck of a lot more. So why would Rowling--or anybody--use a pen name?
I can't speak for Rowling, but I can guess at why she used a pen name. Maybe she thought people would say, "Oh, she's that fantasy writer, she can't write a crime novel. She writes for kids, not adults." Or maybe she wanted to see how her new novel would stand on its own. The novel received favorable reviews from critics before her identity went public. That has to feel good. Her work was judged for its own merit, without the preconceptions her name would have brought with it.
I haven't used a pen name yet. I was too excited to see my name on the cover of a novel, and associated with short stories. But, I plan on using a pen name for various reasons, depending on the books/stories I release in the future.
For one thing, I might not want people who know me to realize what I've published. I don't feel any shame about writing what I write, but some people might look upon certain genres much less favorably than others, and I'd rather avoid any potential fallout.
Then there's a matter of marketing. People might read my fantasy novel and pick up another series, expecting fantasy, only to find it's horror or science fiction or a thriller or a romance. I don't want fans to feel disappointed or duped.
There's also the matter that when readers look at certain genres, they expect certain types of names associated with them. For example, I've never seen a man's name on a romance novel, despite the fact that many romance writers are men. They either write under pen names, or they use initials. I'd love to see a man's name boldly splashed across the front cover of a bodice ripper.
On the other side of the matter, I don't see women's names on the covers of thrillers or hard science fiction as often as I'd like. Again, women write in these genres, but they choose pen names. I wonder, is it the audience that really has these preconceived ideas of who "can" or "should" write certain genres, or does the expectation lie with the marketing departments, or is it some combination of both?