Quick time events vs. timeline-based rhythm games

Using quick time events (QTE:s) is a form of trying to have a cake and eat it too. The game designer wants to control the pacing and drama, but also let the player control the game. The two are difficult to combine, and often the player's control over events is illusory, with the branching storyline soon converging back into one or few options to save production costs. But do we always have to create the illusion of control? As it happens, there's a genre where the players are perfectly happy to play along with completely choreographed action: dancing and rhythm games. Can we combine rhythm and action adventure genres in new and fruitful ways?

Yesterday, I finally got to use my PS3 at work for playing. I spent some time on Dark Souls, Uncharted 3 and Heavy Rain, which got me again thinking about quick time events. At some point, I was wondering that there's not really an analogous mechanic for Kinect games, but actually Kinect Star Wars does have them in the form of "anim GIF" prompts that appear contextually, indicating that one has to kick, jump etc. to proceed.

When playing Heavy Rain, it struck me that it's actually pretty similar to Guitar Hero. Both have action that you play along with in an abstract, simplified manner. In an ideal case, there's an illusion of actually performing a fight choreography or playing the notes of a song, to the extent that a Guitar Hero guitarist can play pretty plausibly on stage at the Video Games Live -concerts, as shown in this video (or actually not shown very well, since the camera is behind the player. I've seen better footage but this is the best I could find right now):

In GH, there's five buttons that represent all notes that one can play. Actual melodies use more, and the meaning of the buttons changes based on the previous ones. Even with just five buttons, one can approximate a melody so that it mostly goes up and down as it should, but the actual intervals are incorrect. If I remember correctly, this actually corresponds to the first stages of the developmental continuum of singing (couldn't check because I don't have access to the full paper from where I'm writing). A video for refreshing one's memory:

In Heavy Rain, you use the right analog stick and shake the controller to approximate the movements of the characters. A video:

The differences between the two

• Heavy Rain, like most games with QTE:s, features some level of branching action based on whether the player reacts to the events on time, whereas in Guitar Hero, the song goes on, and the feedback from misses is implemented through scoring and sound effects. Also, if you make enough mistakes, the song ends prematurely and you get booed out (if I remember correctly - later music games usually let you always finish the songs)
• Guitar Hero and many other music games feature some form of a visual timeline, where one can see multiple approaching events and plan one's action. Action adventure QTE:s usually pop up one by one without such 'prediction horizon'.
• Action adventure QTE:s provide the player a reaction challenge, whereas rhythm games require coordination and precision.

I'm not that fond of QTE:s in fight scenes, because the reaction time is really difficult to tune - there seems to be only a tiny sweet zone between frustrating and boring, and it's different for every player. I'm more of a precision and coordination kind of guy, which brings me to wonder if one could actually have rhythm game style QTE timeline somehow overlaid in an action adventure fighting sequence. This would allow a more rapid pace of fighting, similar to many Hong Kong kung-fu movies:

An alternative to a timeline is to use multiple symbols that appear a predetermined time before the player should act, which most QTE:s do anyway, but in rhythm games the visual display of the remaining time to the correct moment is more prominent, as in Dance Evolution:

The benefit of the approach I'm proposing is that the timing and coordination challenges could be more interesting than the reaction challenge of traditional QTE:s, and only a single choreography without branches would be needed. The obvious problem is to design feedback that makes sense if the player's performance is less than perfect - do we want the "perfect", "good" etc. notifications from dance games or something else? Is the fight over if the player misses too many events? Should there be branching after all, and do we actually need two branches per event, one for being too late and the other for acting too early.

As a Google search will tell you, there's already many games with both rhythm and fighting elements. However, fighting to the beat of music as in this Kick Beat gameplay video or in Rhythm Fighter is a bit too much for me.

I really like the choreography, music and soundscape in Uncharted 3 and Heavy Rain. I just wish I could see more than one event in advance to get into the flow of the choreography and still have a reasonable challenge, similar to reading a note sheet when playing an instrument. This is what, e.g. Dance Central and Guitar Hero do, although due to the complexity of the player's realistic movements in DC, the choreography is not fully visualized on a timeline. Instead, one sees a timeline of flashcards, which have to be memorized in the training mode.

For Uncharted 3 and Heavy Rain, I think the action abstraction level is pretty much correct, but I guess I just feel strange trying to react quickly to button symbols shown on screen instead of the opponent's actual actions. The translation of symbols to actions feels more natural when given more time and when seeing multiple symbols at a time, maybe because the relations of the symbols to each other (e.g., the intervals between musical notes) help in decoding and executing the correct the motor sequence.

If anyone reads this and knows about this kind of experiments, please let me know.

Additional thoughts: The illusion of playing a guitar or fighting with QTE:s is not there at first, but at least it happens with me once I've become accustomed to the interface, I don't have to think of the real-world actions, I can focus on the results and feedback, and I get the timing right. Principles at work: We project ourselves to the avatar, our motor system assimilates the game interface through repetition, correlation is perceived as causation, our sensory integration system perceives two events as one if they happen close to each other in time. The same assimilation of the interface etc. happens of course with all action games, but the interesting thing is that it happens despite the player not having control or initiating the actions.

About dance game timeline displays and other choreography visualizations: The visualization of full-body movement over time in a compact and intuitive manner remains a research challenge, although various notations have been developed.