Nihonjin no Shiranai Nihongo (The Japanese that Japanese People Don't Know) is a collection of real-life stories centering on a young Japanese woman as she teaches Japanese to a group of foreigners in Tokyo.
Presented in manga form, the stories are usually about the simple questions her students ask her and the culture shock she experiences in trying to answer them. What comes out of these stories is an interesting look at why the Japanese language is the way it is and how it affects the cultural mindset of the Japanese.
In one strip, an American student is crushed to see so many wrong answers on his last test, only for the teacher to explain that a circle means that the answers are correct. What follows is an explanation of how Os and Xs are viewed in countries around the world, eventually leading to the PlayStation controller and its shape-labeled buttons.
Because in the Japanese mindset X means "no, bad, wrong, or cancel" it is the perfect label for the PlayStation's "back button" in the home market. But outside of Japan, there is no implicit negative association with "X." So when Sony swapped the default functions for the X and O buttons for the PlayStation's international release, they probably thought there would be no problems caused by the change.
However, when Western games come to Japan, the game buttons are often still mapped to the international format. Thus, when Japanese people try to play Western games, they can't get used to the idea of X meaning "yes" and O meaning "no." To put it another way, it'd be like going to another country where on the traffic lights red means "go" and green means "stop." For the Japanese, the Western controller layout is more than a change of button mapping; it's something fundamentally culturally counter-intuitive.
If you're wondering why the full set of shapes and colors were chosen for the PS1 controller, PlayStaion designer Teiyu Goto had this to say in a 2010 interview with 1up:
I gave each symbol a meaning and a color. The triangle refers to viewpoint; I had it represent one's head or direction and made it green. Square refers to a piece of paper; I had it represent menus or documents and made it pink. The circle and X represent 'yes' or 'no' decision-making and I made them red and blue respectively. People thought those colors were mixed up, and I had to reinforce to management that that's what I wanted.
And while there isn't any official reason why Sony swapped the X and O buttons, maybe it's as simple as the old saying: "X marks the spot."
Update: Added info about Goto's design choices for the PS1 controller.