This is what I told her/them:
Tell him not to go the vanity publishing route. If he's written a good book he can find a real publisher for it, and it's a crime to do vanity publishing. Tell him to remember that a legitimate publication scheme means money always flows toward the author.
Also tell him that it is in fact possible to get an agent without being published. It's a lot of work, but so is everything in the publishing industry.
And you're right: he needs to send it to publishers and, probably more specifically, agents who market things in his genre. All stories cross genres: practically every book you'll read has a romance, and I use mysteries for the backbones of my fantasy novels all the time. What he needs to do is determine what genre is most like his, and the answer to that may be "mainstream fiction" (that's what it sounds like to me). He needs to get a Writer's Market and a guide to literary agents (the book paired with that, 'Agents Directory', is actually by someone who used to work for the DMLA, my agency), and he needs to pick up books by authors he enjoys reading/whose work is similar in some fashion to his, and see if any of them thank their agents in the acknowledgments; that's a very good way to find agents who are actively working in your genre.
A hugely useful resource for legitimate agents and agencies is Preditors & Editors. It's also got query letter suggestions, proposals, synopses--it's a really valuable resource.
He needs to come up with a one sentence pitch for the book--an elevator pitch. My book in 15 words or less, that sort of thing. My pitch for URBAN SHAMAN was:
"A Seattle cop with no use for the mystical has a near-death experience and is offered a choice between dying, or life as a shaman. When she chooses life, she finds herself neck-deep in a murder mystery and up against a couple of old Celtic gods."
Something like that. There's a whole bunch of ideas at ForwardMotion.
He'll need something like that for a query letter, but first he needs to study agents and agencies and find out who would be a good match for him.
He should send out query letters, FOLLOWING THE AGENCY'S INSTRUCTIONS on what they ask for in a query letter (some just want a letter, some will ask for the first 5 pages of the manuscript, etc), and just keep trying until he gets a hit.
Where does he live? One of the best ways to get an agent's attention, or an editor's, is to go to writer's conferences. Lots of them have editor/agent appointments where you can pitch (which is why you need to have that elevator pitch memorized!), and they'll often ask you for your book, or at least a proposal on it, at those things. That's one way to get past the unsolicited submissions problem that many publishers have. I can't recommend highly enough going to these things--the Rocky Mountain Fiction Writers' conference in Colorado every September is what got me actually on the path to publication. I would hugely recommend that one, if it's possible for him to attend.
Pretty much the last thing I can say is not to get discouraged. This is just not an easy business to break into, but if he has in fact written a good book, then he'll make it. Publishers want good books. They really do. And most of what they see is absolute crap, so if you've genuinely written something good, then you're already way ahead of the game.
Ok, no, I've got one more thing to say after all. :) You're right: I spent a whole lot of years honing my craft and getting to the point where I could sell. It didn't actually take me that long to sell, once I'd gotten there, but I was goddamned methodical about it. This is not a crap shoot and it's not about luck. There's some aspect of that, sure, but it's a business, and you need to treat it like one, even if all you're doing is trying to break in.
Polish your manuscript. Make it the best thing you possibly can. Find the agents you most want to be represented by. Query them.
Start writing your next book.
There's a hell of a lot of down time in this business. An awful lot of wait time. Do not spend that time fretting. It's useless. Do what you can--query, query, query, until you get a hit--but don't wait til you've hit it big with the one book to write the next. One of the first things an agent who takes you on is going to say is, "What else do you have?" You want to have something. URBAN SHAMAN was the fifth manuscript I wrote, and I'd written 7 by the time I sold it. That's paying off for me in a big way now--I've been able to sell that series and two others since then because I had either a complete manuscript or a significant partial manuscript completed on those other series. Use that time, because right now it's all you've got on your side. :)
Good luck to him. :)