According to GameInformer, the combat went through several variations: "The first one being a full rhythm action game! The second one was prototyped in 2D, which popped up whenever you got into a fight, and involved colored circles bashing into each other. This actually formed the basis of the final system."
Rhythm...action...game? THANK YOU FOR NOT USING THAT COMBAT SYSTEM. More factoids and tidbits in the link below.
Tidbits: What You Didn't Know About Batman: Arkham Asylum [GameInformer via Joystiq]