Once upon a time, Japanese games eclipsed Western ones. They looked better, were more fun and were just better. Those days have passed.
At last year's Tokyo Game Show, Mega-Man designer Keiji Inafune said that the Japanese game industry was "finished". Inafune, it seems, was speaking in hyperbole. The Japan industry is not finished — case in point, Nintendo. That's a Japanese game company that has been doing extremely well.
The rest of the Japanese industry, however, has been hit pretty hard. The Western market has exploded, and Western game development has truly come into its own.
2ch, the largest bulletin board in the world, had a thread on this exact issue on why Japan is so behind the curve on game development. The thread is of interest as it provides a look at what those in Japan think about the issue. Game translator and Yokai Attack! author Matt Alt translated the replies:
-Japanese game developers are born and raised and live in Japan. With games, with art, with otaku culture, and more, we have a distinct background and history. In the future we're going to have to make that a sales point, or find a way to "mine" it. For example, we'll never beat Asia when it comes to the cost performance of rendering realistic-looking graphics. But when it comes to "stylized" (deforume) stuff we can't be beaten.
-Abroad, university grads, grad school grads, and even PhDs make games. The vast majority of Japanese game industry people have a university degree or some game-design school diploma at best. There's no way we can win.
-Japanese game scenarios are the pits. We may be behind in graphics but that isn't a problem so long as they're fun and interesting. The problem is that teams can't focus when they have to carry through a crappy scenario to its end. I don't know whether to describe them as "childish" or "otaku-oriented" or what but a lot of games are just too ludicrous for adults to appreciate.
-A lot of foreign game companies hire pro screenwriters. Here we leave this critical part of the process to directors and programmers and other amateurs.
-The PS1 era had a lot of cool and lighthearted games, but starting with the PS2 "otaku games" became the norm.
-I find it really strange that in an era when we know foreign gamers love first-person shooters that not a single domestic company has even tried to make one.
-The problem isn't a lack of education or technical skill. Believe me there are plenty of over-educated dudes in this industry. The problem is the total lack of ability of management. But you could say that about Japanese businesses across the board.
-We may be behind the curve in some respects, but answer me this: is there any country out there that can beat Japan when it comes to ero-games?
There are certainly some valid points in these comments, but I don't think it's simply a matter of hiring pro screenwriters. The big issues with some (not all) Japanese games are things like interface, controls and save systems — among other things. Fix that stuff! And all that is totally manageable.
Besides that, Japanese developers should really do what they have always done: make games for Japan. It has served the country well in the past. No reason why it couldn't in the future.