Breaking into mainstream publishing is a difficult process, but I think not for the reasons that many people assume. I think there's a fundamental disconnect between aspiring authors and publishers that needs to be addressed: that publishing is a business, and books are the product. At a basic level, the author is producing a product. It could be a very good, well-written product, one that they've spent years perfecting, but it is still a product, and it needs to have some kind of market appeal. It has to be sellable, or you have to make it sellable. This is the reason that high literary works won't do as well as Harry Potter. The boy wizard story appeals to everyone, whereas something highbrow may only appeal to a select demographic, even if the critics love it.
Do you need an agent? Yes, mostly. They're not just there to take a percentage of your earnings - their business is to know about the industry and to be a guide and mentor to you, the creator. Their service is to get your book to the right people, negotiate the best deals, and help you create the most marketable, sellable product you can. Your job is the maker; they are your salesperson.
You can go it alone, if you're comfortable doing so. Plenty of authors like to concentrate on nothing but the writing, or they have day jobs. They need agents.
First thing you need to know is that publishers want to say yes to your book. Does that sound weird? It's true - they want to say yes to you, and buy your book. They want your book to knock their socks off, because they want to buy books that knock people's socks off. Books like that sell thousands of copies and make lots of money. The problem is there could be any number of reasons why they have to say no, and many of them don't have anything to do with your book.
- They've already bought a book with a similar theme recently
- They don't think it'll sell widely enough
- They've already bought enough books total this year
Anything, really. They're a business. What they want is a book that will sell lots of copies and make lots of money for everyone. If they don't think your book will do that, they're not going to say yes - and you should thank them for that, because it means they won't go out of business and you can try again with another book at a later date.
I wish I could just keep repeating this to everyone I meet who mentions they want to be a writer - It. Is. A. Business. It's a profession. You have to view it as a job and treat it seriously. And it's never, ever easy.