Personally, when it comes to game publishing, I think a lot of the failures arise from individuals attempting to over-reach the market. You hope you’ll get a huge market, but there is no guarantee, and in case you do, you have to prepare for it just in case. But why is that? Gaming is a very personal sort of hobby and most marketing occurs through word of mouth, so with that being the case, why not publish small in a way that can expand rapidly if necessary? I’d suggest Lulu.com with a markup, for example. Not -all- of their books are beautiful, but they do replace the ones that arrive poorly made. The whole infrastructure exists and their materials can be marketed through Amazon in both ebook and hardcopy form–the entire publishing structure for a few hundred dollar gamble.

For example, I write and do game design–both in pre-existing game systems (Pathfinder) and with my own gaming systems–but the reason I don’t try to publish anything is because I don’t want the longterm hassle of royalties, everyone taking their piece of the pie before me, having to hold myself to someone else’s standards… etc. Mostly because I’m hard enough on myself as is, and I think of the games as being equal parts function and art (beyond the actual pictures of course). Because of this attitude, I don’t get along with a lot of editors and I prefer to do everything, typesetting included.

For people who have their hands in every single aspect of their game–in almost a boutique style in fact–it makes more sense to publish on demand until demand is great enough to warrant a larger printing.

If someone threw a conglomerate of writer/designers together and took submissions for entry (to prove you warrant the mutual co-op as it were) before doing a small-scale publishing effort through a third party company like Lulu (it’s not like gaming books don’t already cost in excess of $30 most of the time anyway) and split the profits evenly among the co-op, I’d join it. That simple. I would take one of my better pieces–perhaps my whole mechanic system–and feed those pieces to the gristmill to see what comes out. At least that way, I’d have creative control through the entire process, with the help of other people that want it to succeed, but don’t have a pre-existing $10,000 debt riding on it from the printers.

I think there is merit to both of your options… I would definitely not sacrifice either of them, however, just because one is harder.