Math libraries for c++

Stumbled upon this useful article: http://www.mfoot.com/blog/2012/01/23/choosing-a-maths-library-for-cross-platform-c-game-development/

I've been using OpenCV for generic linear algebra, and then some others for fixed size 3D matrices and vectors. However, I've grown tired of OpenCV's quirks, especially the difficulty of debugging due to its internals, where void pointers are passed around and a custom RTTI system tries to make sense of the data (legacy of the first OpenCV versions that were C only). It seems to be fairly easy to cause an exception for which one doesn't even get a stack trace. Maybe it's time to switch to Eigen, which is apparently equally useful for large and small matrices, and also performs faster than GLM.

Edit May 27: Eigen definitely seems to work well and is easy to use. Creating math-heavy applications in C++ just got so much easier.

Edit June 12: I'm a full Eigen convert now, preaching to everybody who cares to listen =) Many matrix size and alignment errors are catched at compile time, and the API is flexible - for example, one can declare a matrix m and then say, e.g., Vector3f v=m.rowwise().sum(). More in this API showcase.

Beyond Ragdolls (we're hiring)

I'm proud to share the first public results from our Future Game Animation research project, supported by 7 game and animation companies and Tekes (Finnish Funding Agency for Innovation). Our paper "Online Motion Synthesis Using Sequential Monte Carlo" was accepted to this year's ACM SIGGRAPH conference.

Note: we're hiring! If you'd like to do a doctoral or Master's thesis about character motion synthesis using physics simulation, email me at perttu.hamalainen@aalto.fi.

Here's the video accompanying the paper:


Here's also some image sequences:

Executive summary: We've developed an "intelligent ragdoll" that can dodge projectiles, land on its feet, and get up after falling. All movement is emergent and does not require a library of animation or motion capture data. The goal is to eliminate animation-related delays in game prototyping and development, and enable new forms of gameplay.

I've previously posted a Unity script for turning a Mecanim character into a ragdoll that can get up after falling. While the script is quite simple, the resulting motion does not respect physical constraints like non-penetration, and the script also needs the get up animations. Using considerably more code and CPU, our research system - also integrated with Unity - can generate getting up and other behaviors without any animation data, by optimizing the control parameters of physics simulation. This will probably be how intelligent characters are implemented in future games, although we still need more work on computational efficiency and movement quality. We will be working on this at least until the end of 2015. In addition to improving the technology, we also plan to create some proof-of-concept prototypes of novel gameplay that could be created with it.


FAQ:

Q: Isn't this like NaturalMotion's Morpheme/Endorphin?

A: Yes and no. What I've seen from NaturalMotion (e.g., Clumsy Ninja) is more like my Mecanim ragdoll script in that it uses a lot of premade animation. While press and NaturalMotion's marketing like to portray it as "one big simulation", Clumsy Ninja's kicks, trampoline jumps, getting up etc. behaviors are premade animations. There's some degree of procedural footstep generation when dragging the character, but also artificial, non-physical glitches and sliding of the character, e.g., when the character adjusts its distance to target before kicking or punching.

I'm not saying that Clumsy Ninja isn't a great product. However, creating such believable and intelligent characters is presently highly expensive due to the amount of custom motion capture, animation, and scripting. The main point of our work is that it requires no motion capture, animation, or precomputation and is fully based on physics simulation.

Q: What about the recent viral video where characters were walking?

A: The video (by Geijtenbeek et al., who we do cite in our paper and have enormous respect for) shows the results of offline optimization of a controller that can generate walking in real time. The drawbacks there are that 1) the offline optimization takes a lot of time and 2) the controller is motion-specific, and the characters can't, e.g., get up after falling (which also applies to NaturalMotion's procedural balancers used in Endorphin). Note that the boxes thrown at the characters in the video appear very light and the steering and ground angle variations are quite modest. Our system does not require offline optimization or other precomputation and produces a wider range of movements, although the quality is not as good (we're working on that...).

Q: Can I download the code?

A: The system is presently quite complex and very much a work in progress. We're planning on an open source release sometime in 2015.

Augmented climbing and HCI in sports

In this year's ACM CHI conference, we had three submissions: a video showcase and a poster/extended abstract about our augmented climbing wall (a project of Raine Kajastila, a post-doc in our group), and a HCI in sports workshop paper about designing for empowerment in full-body human-computer interaction.

Unity and version control

Here's some info of version control in Unity for this fall's game project course. Summary: it's ok to use git, svn etc. if you know a few tips, and Aalto doesn't have licenses for Unity's own version control system (the Asset Server). Only add the Assets and ProjectSettings folders to version control. Don't add the automatically generated Library folder or the monodevelop and visual studio solution files.

Also: DON'T SWITCH BACK TO UNITY WHILE YOU'RE GETTING/UPDATING FILES FROM VERSION CONTROL. This will cause major errors, as Unity starts importing and reimporting the files right away, and in the process it may create new .meta files if the correct .meta file was not yet updated.

Basic info about setting up external version control: 

http://docs.unity3d.com/Documentation/Manual/ExternalVersionControlSystemSupport.html

Some additional points:

• If you are using Unity Pro or Academic (e.g., school computers) go to project settings/editor and set asset serialization to "force text", which enables svn, git etc. to merge the files. 
• However, the merging doesn't always work, and to minimize conflicts, you should avoid modifying your scene files. The way to do this is to have everything as prefabs and edit the prefabs instead of modifying the scene. This way, it's less likely that several people modify the same file.
• Enabling external version control as explained in the document above creates a .meta file for each file. Always remember to add the metafiles to your repository. The metafile contains the unique identifier (GUID) that Unity uses, e.g., when you link a prefab to a script's public GameObject property. If you don't commit a metafile to the repository, it will be regenerated on each person's machine with different GUIDs, which breaks the object linking and you get null reference errors.
• Be careful when moving and deleting assets. If you delete or move assets inside the Unity editor, svn etc. will of course bring them back when you update before committing your changes. However, once the metafiles are in use, you can safely move and delete files outside Unity as long as you move and delete the metafiles too. Unity will recognize the moved assets as the same ones based on the metafiles.
• When googling to check if there's a good tutorial or if I missed something above, I noticed there's quite a lot of "free version of Unity doesn't support version control". However, things have changed and as of version 3.5., metafiles are no longer a pro-only feature.
• The "force text" option is apparently a pro-only feature, but there are workarounds: http://answers.unity3d.com/questions/8740/version-control-workflow.html & https://github.com/terravision/UnityTextScene

Unity and FMOD Studio integration

Edit 12 June 2014: This post was based on the very first FMOD Unity integration from fall 2013. There's been changes and the project file should probably be updated. Please (also) check some more recent tutorials.

There's finally an official FMOD Studio integration for Unity, available from http://www.fmod.org/download/. It has just been released and contains no Unity examples, so I prepared a bare bones Unity project and an example FMOD Studio project that you can download from here https://www.dropbox.com/s/vyk29kyhmvm22dc/part6_sound.zip

The main reason to use this is that it enables your sound designers to edit, mix and master your game audio using the FMOD Studio interface while the game is running. Using FMOD studio, you can also create, e.g., a footstep sound event that will randomize the pitch and selected sound sample without the game having to know anything else than the name of the event. Using FMOD is free for non-commercial purposes and the license prices are also pretty reasonable for commercial work.

Most commercial games with decent sound design use such live mixing and centralized sound management approach, either using FMOD, WWise or some custom tools. Unity does use fmod for its own sound, but it does not provide any mixing tools and a sound designer will go insane if he/she has to manage hundreds of sound objects in the Unity editor. The main limitation of the FMOD Studio integration is that it only supports iOS, Android, Windows, Mac but no web player. If you want your sounds to work in the web player, I suggest you check out the Clockstone Audio Toolkit that provides mixing and management on top of Unity's internal sound system.

Here are the steps you need to add FMOD Studio support to your Unity project. These instructions can also be found in the readme.txt of the .zip above.

1. Import the fmodstudio10203.unitypackage from this folder. You can also check for a later version from fmod.org. To import in Unity editor, select Assets/Import Package/Custom package.
2. build your FMOD banks (File/Build)
3. In Unity, select FMOD/Refresh Event List and select the GUIDs.txt from your FMOD project's Build folder. Note: you need to do this whenever you rebuild the FMOD project. This will create FMODAssets and StreamingAssets folders in the Unity project and copy the FMOD banks to the StreamingAssets folder.
4. When your scene starts, load a bank (see the Start() method in main.cs)
5. Use the FMOD_StudioSystem helpers such as PlayOneShot() to play sound. See the OnGUI() method of main.cs. You might also have to use the lower level API (fmod.cs, fmodstudio.cs), as the unity integration package has been released very recently. The integration package doesn't contain any examples, and for example code, you should consult the c++ examples in the FMOD Studio API install, which you  can download from fmod.org.
6. To enable live mixing using FMOD Studio, uncomment the first line of FMOD_StudioSystem.cs, found in the plugins/FMOD folder. Also check the "Run in background" checkbox in Unity Editor. You'll find that by selecting Edit/project settings/player and then selecting the Resolution and Presentation panel in the Inspector. Now press play in unity, select File/Connect to game in FMOD Studio and click ok. You can now adjust audio event properties on the fly while the game is running. 

Unity Mecanim and ragdolls

Update 17 May 2014: If you're interested in physics simulation and character animation, check also our latest research. Also, please note that Mecanim has had some changes since the original post, and now the script could be implemented more elegantly. I also haven't tested the script with different characters - the comments suggest that some characters with different bone structure might not work out of the box.

Sharing a Unity project with a RagdollHelper.cs script that can be used to turn a Mecanim character into a ragdoll and then back. Using the script is simple: there's just one public property called "ragdolled", which can be set to true or false. When the property is changed from true to false, the script blends from the ragdolled pose to a get up animation. This is illustrated in the video below.

Transitioning from Mecanim animation to a ragdoll is as easy as disabling the Animator component of the character and setting all ragdoll rigid bodies to non-kinematic. The reverse is more complex, and my solution is not without kludges, as Mecanim doesn't allow as low level control of the animation as Unity's older animation system. The main problems are that one can't use scripting to enforce an immediate state transition without a blend, and apparently one can't also add states programmatically into an animation controller.

In the old animation system, one could create a new animation clip from the ragdoll's current pose, turn of ragdolling and blend the animation from the created clip to a recovery animation. In Mecanim this is not possible, and after turning Mecanim back on, we have to manually lerp the hip position and slerp all body part rotations back towards the last ragdolled pose in LateUpdate() to achieve a smooth blending. Additionally, we have to wait some frames for Mecanim to transition to the get up animation while using LateUpdate() to move the pose back to the ragdolled one, and then read back the get up animation start body orientation and position, and update Mecanim character's root so that the animated orientation and position match the ragdolled one as well as possible. In the old system we could just update the character transforms with a single script command according to a specific frame of a given animation, and then read whatever info we need. Actually we could possibly use the legacy animation system for that, but then the script would need additional handles to the animation etc., so it seems there's no really simple and beautiful way. One could also perhaps use an editor script to precompute the info.

Here's the Unity project as a zip file in case someone finds it useful: https://drive.google.com/file/d/0B8lAtheEAvgmYUlqX1FjNm84cVU/view?usp=sharing&resourcekey=0-kVdY0_KU-BRQP9xZibN0Fw

Open the test scene, click with mouse on a body part to add an impact to the character, press space to make it get back up.

Update 5 Dec 2013: Today I noticed that Unity 4.3 has added the Animator.Update() which could be used to implement the ragdoll to animation blend more elegantly without the LateUpdate() trickery. I'll have to update this example when I have time.

Homepage

Restored and updated my old research homepage. http://perttu.info is now redirecting there. Nothing special, mainly lists of publications and projects.

Game programming tips

Starting a link collection here...

Remember to avoid transform.position=xxx... in Unity. Here's why:

http://answers.unity3d.com/questions/9893/which-getters-are-hiding-expensive-side-effects.html

Trampolines as a business

Related to our Kinect trampoline and circus games, I just checked that Googling "trampoline park" gives 628000 results. Trampoline centers/parks seem to be a growing branch of the family entertainment center business, as discussed, e.g., in this article about the Sky Zone franchise, which had 15 centers in the U.S. and planned to more than double that when the article was written in 2012.

Here's two pretty compelling videos, the first one from House of Air from the U.S. and the second one from Bounz, one of the few European ones.

Despite the thrill, it remains unclear whether trampoline parks are craze that will fade like the soft-contained-play (SCP) and inflatable centers did, as explained in this article. According to the author, trampoline centers are similar to SCP and inflatable centers in that they too haven't proven their business model for long-term viability, and they are at the same time more dangerous. However, I think that while trampolines may be more dangerous than bouncing castles, they also present much more potential for long term excitement, as the increased air time and softer landings make it possible to learn a wider variety of skills and tricks. For example, it's not very comfortable to land on one's back or belly on an inflatable bouncing surface, whereas it's quite easy on a trampoline and one can also do many interesting variations, such as turntabling:

Action Games Workshop info (Aalto course code 25447)

Since we are using a generic course code and I can't edit the contents and learning outcomes in Noppa, I'm posting the material here.

Contents

Many games are simplifications and abstractions of reality. The designer's job is to abstract away irrelevant and/or cumbersome details, and choosing what to focus on is not trivial. Background research helps, but in addition to gathering mediated experiences in the form of text, images, and video, it's useful to experience things in person.

On this course, we will experience various sports and their digital representations to identify yet uncharted possibilities for action games. The course consists of physical activities mixed with lectures, gaming sessions, and interaction sketching exercises. In spring 2013, the course is organized for the first time as a workshop including parkour, trampolining, archery, swordfighting, contemporary dance, and bouldering (climbing) with help of professional instructors. 

An additional goal of the course is to share the latest insights from Aalto’s games and motion related research. 

Learning outcomes  

Gain insights and ideas for gameplay design from the points of view of, e.g.,  embodied experiences, biomechanics, motor learning and performance, physics simulation, and virtual cinematography. Understand the variety of embodied experiences, ranging from passive media to console games to experimental mixed reality games. 

How to register

You can sign up in Oodi: https://oodi.aalto.fi/a/opintjakstied.jsp?Kieli=6&Tunniste=25447&html=1.

Click "register" at the bottom of the page. Note that the Oodi page is a generic one, since the course doesn't yet have its own course code and page. Note also that you can find the updated course description by clicking the "Workshop I" teaching event at the bottom of the page, but Oodi doesn't let you register on that page - I would have linked directly to it otherwise.

The number of participants is limited to 14, not including me and Miikka of course.

Prerequisites

Preliminary exercise: for the first gathering, prepare a few minutes presentation about a physical activity, such as your favorite sport. Illustrate your presentation with image(s) or video. Answer these questions: What's interesting, fun and motivating about it? How do you feel when practicing/competing? Has it affected the way you experience the world? How? Is there a digital version of it or a digital dimension to it? If so, how does the digital user experience differ from the real thing or what does the digital dimension contribute?

Schedule

The course is organized during 22-26.4.2013. We'll start each day with a physical activity.

Monday 22nd 9-11am: Trampolining and parkour at Taitoliikuntakeskus. How to get there: http://www.taitoliikuntakeskus.fi/?q=node/15

Tuesday 23rd 9-12am: Archery at Archery Club Wilhelm Tell. Place: Katri Valan puiston väestönsuoja: http://bit.ly/16Ivuva

Wednesday 24th 9:30-12:30: Medieval swordmanship at The School of European Swordmanship. Address Luiskatie 8, Helsinki (e.g, bus 77)

Thursday 25th 9:30-11:30: Contemporary dance at Helsingin Tanssiopisto. Address: Kaikukatu 4, http://bit.ly/Zi5LoQ

Friday 26th 10-12am: Bouldering (climbing) at Boulderkeskus, Konala. Address: Ruosilantie 1, http://bit.ly/16IknEq 

Remember to bring indoor sports clothes and shoes. Climbing shoes will be provided by Boulderkeskus.

In addition to the activities, we'll have lectures and exercises each afternoon from 13:00-16:00 at the Media Lab games classroom. I'm currently preparing the materials, and so far the tentative themes for the afternoons are controls & virtual cinematography, character animation & biomechanics, physics based gameplay, story & emotion, and motion games & games for learning.

Visualizing Fitness Function Landscapes

I'm currently preparing a project about game animation tools and technology. The project is exploring and leveraging technologies like physics simulation, optimization and machine learning. If you are interested, the recent review by Geijtenbeek and Pronost is a good starting point.

Related to the optimization of the control parameters of physics simulation, I'm trying to form an intuition about the properties of the related fitness function landscapes. It's impossible to accurately visualize the landscapes of high-dimensional biped motor control and planning problems, but here's a simple 2d physics example: throwing a ball in 2d with the angle and speed as the optimized variables, and allowing damped rebounds from the ground so that the ball trajectory is not just a simple parabola. This isn't exactly rocket science, but I wrote the code to as an exercise in interactive visualization in Matlab, which I stopped using while working outside Aalto.

It was my first time I tried to do implement the "up and down the ladder of abstraction", that is, letting the user explore a summary view (the fitness landscape) to browse concrete examples (the ball trajectories). There was some non-intuitive searching for the right places to put the getCursorInfo(), pause() and subplot() commands to prevent the display from freezing - hope this helps if anyone wants to do the same. I also tried to implement the UI using Matlab's linked data, but couldn't get the display refreshed.

Edit 29.1.2013: I was asked why there's peaks where there should be a continuous ridge. It's apparently because of the large timestep that causes a simulation error that varies as a function of how the floor contacts align with the timestep grid. It also depends on how the contact handling is implemented. The Matlab code below uses a smaller timestep and produces a more continuous ridge.

Here's the Matlab code:


minVel=1;
maxVel=5.0;
minAngle=-0.5*pi;
maxAngle=0.5*pi;
timeStep=0.02;
nSteps=100;
startPos=[0,1.5];
velSteps=64;
angleSteps=64;
E=zeros(velSteps,angleSteps);
target=[2, 0.5];
curve=zeros(nSteps,2);
curves=zeros(nSteps,2,velSteps,angleSteps);

%step over all ball launch speed and angle combinations
for velStep=0:velSteps,
    speed=minVel+(maxVel-minVel)*velStep/velSteps;
    for angleStep=0:angleSteps,
        angle=minAngle+(maxAngle-minAngle)*angleStep/angleSteps;
        pos=startPos;
        vel=[speed*cos(angle), speed*sin(angle)];
        closestDist=10000;
        closest=[0; 0];
        closestTime=0;
        
        %simulate the ball trajectory for this launch velocity
        for n=1:nSteps,
            %save this trajectory point
            curve(n,:)=pos;
            
            %update closest to target
            dist=norm(pos-target,2);
            if (dist<closestDist)
                closestDist=dist;
                closest=pos;
                closestTime=n*timeStep;
            end
            
            %physics update
            pos=pos+vel*timeStep;
            vel(2)=vel(2)-9.81*timeStep;
            
            %bounce from floor
            if (pos(2)<0 && vel(2)<0)
                timeInsideGround=pos(2)/vel(2);
                vel(2)=-0.7*vel(2);
                timeAfterBounce=max(0,timeStep-timeInsideGround);
                pos(2)=timeAfterBounce*vel(2);
            end
            
        end    
        %store the whole trajectory
        curves(:,:,velStep+1,angleStep+1)=curve;

        %update fitness
distanceThreshold=0.1;
        fitness=0.5+0.5*tanh(-(closestDist-distanceThreshold)/0.05);

        %soft threshold for hitting time
        timeThreshold=1;
        fitness=fitness*(0.5+0.5*tanh(-(closestTime-timeThreshold)/0.05));
        
        %store fitness function value for these params
        E(velStep+1,angleStep+1)=fitness;
    end
end
%plot the fitness surface
subplot(1,2,1);
surf(E);

%get the handle for querying data cursor info about the fitness
h=gcf;
dcm_obj = datacursormode(h)

%plot a trajectory of the ball
subplot(1,2,2);
curve=curves(:,:,1,1);
plot(curve(:,1),curve(:,2));

%in a loop, keep plotting the ball trajectories
pause on;
while 1
    info_struct = getCursorInfo(dcm_obj);
    if size(info_struct)~=0    
        curve=curves(:,:,floor(info_struct.Position(2)),floor(info_struct.Position(1)));
        subplot(1,2,2);
        plot(curve(:,1),curve(:,2));
        axis([0 4.5 0 3])
        hold on;
        plot(target(1),target(2),'c+:')
        hold off;
    end
    %have to pause to let the display refresh
    pause(0.01);
end

Game marketing resources

To help our students in promoting their games, I did about an hour's worth of searching, both using Google and digging up old stuff from my brain. Dumping the results here for future reference, roughly in the order of importance.

About creating press releases etc.

http://gdcvault.com/play/1014971/Indie-s-Got-PR-Talent

http://arstechnica.com/gaming/2011/12/how-to-market-your-indie-games-ben-kucheras-lecture-at-run-jump-dev//

http://gamasutra.com/view/news/167706/Ask_Gamasutra_How_to_annoy_a_games_journalist_with_a_press_release.php#.UQE7syfZZ8E

http://www.gamestyleguide.com/VideoGameStyleGuideeBook.pdf (Talk to the journalists using their own language)

http://dopresskit.com/  Presskit making tool. Note: I haven't tried it personally.

How to make good trailers

http://indiegames.com/2012/04/ask_indiegames_what_makes_an_e.html

http://blog.kertgartner.com/2012/03/making-entertaining-and-engaging-video-game-trailers/

http://www.indiegamegirl.com/how-to-make-a-video-game-trailer-kick-ass/

http://www.ign.com/articles/2012/05/01/five-sure-signs-of-an-awesome-game-trailer

http://www.reddit.com/r/Games/comments/16g1x7/that_great_cyberpunk_2077_teaser_got_me_thinking/

http://www.reddit.com/r/Games/comments/zd84h/what_have_been_the_best_game_trailers_you_have/

http://gametheoryonline.com/2011/05/06/video-game-marketing-creating-game-tr

Stats to inform your decisions: http://engage.tmgcustommedia.com/2011/04/101-online-video-stats-to-make-your-eyes-glaze-over/

A note on backstory and child's play

The backstory is perhaps the most common literary device used in games. Understanding its importance is easy if one has followed the mobile game space, where many hugely successful games have a story-based motivation that can be expressed in a single sentence, such as "The frog wants the candy" or "The birds are angry because the pigs stole their eggs".

What wasn't obvious to me until I became a dad was that the backstory works perfectly similarly in child's play. It's been amazing to see firsthand how big a difference it can make in motivating play. For example, when its -10 centigrades outside, a common temperature this winter, my son hates to go out because putting on all the layers of clothes is such a tedious process. However, he immediately jumps up from the sofa if I throw in a backstory.

Me: "Let's go outside sled riding"
Son: "I don't want to go outside. It's boring. I don't want to wear the overalls."
Me: "But we can pretend to be Angry Birds. You can be the laser bird and have a snowball as a laser."
Son: "Cool! Let's go right away! Come on, dad!"

Malone's and Lepper's taxonomy of intrinsic motivation comprises four main motivating factors: challenge, curiosity, control, fantasy. Having a backstory provides a fantasy setting, and also provides material for curiosity, because one can explore how far the fantasy can be extended by using the play space, e.g., the snowballs. It can also inspire the child to come up with challenges. Many times, if I'm too tired to play cops and robbers, my son is happy to do all the running around by himself while I just sit on the couch and press imaginary buttons to control him on an imaginary iPad.

"Now I'm in level 13, and this is really hard. There's dynamite and I must not fall into the water (floor). Now press the button that makes me run to the door."

"You are the laser piggy. I must not hit the lasers."

Pretty contradictory considering that digital games are supposed to promote a sedentary lifestyle. Of course my son would never let go of the iPad voluntarily, but allowing him a controlled amount of daily gameplay seems to only boost his physical play by providing new backstories, similar to the books we read.

Quick time events vs. timeline-based rhythm games

Using quick time events (QTE:s) is a form of trying to have a cake and eat it too. The game designer wants to control the pacing and drama, but also let the player control the game. The two are difficult to combine, and often the player's control over events is illusory, with the branching storyline soon converging back into one or few options to save production costs. But do we always have to create the illusion of control? As it happens, there's a genre where the players are perfectly happy to play along with completely choreographed action: dancing and rhythm games. Can we combine rhythm and action adventure genres in new and fruitful ways?

Yesterday, I finally got to use my PS3 at work for playing. I spent some time on Dark Souls, Uncharted 3 and Heavy Rain, which got me again thinking about quick time events. At some point, I was wondering that there's not really an analogous mechanic for Kinect games, but actually Kinect Star Wars does have them in the form of "anim GIF" prompts that appear contextually, indicating that one has to kick, jump etc. to proceed.

When playing Heavy Rain, it struck me that it's actually pretty similar to Guitar Hero. Both have action that you play along with in an abstract, simplified manner. In an ideal case, there's an illusion of actually performing a fight choreography or playing the notes of a song, to the extent that a Guitar Hero guitarist can play pretty plausibly on stage at the Video Games Live -concerts, as shown in this video (or actually not shown very well, since the camera is behind the player. I've seen better footage but this is the best I could find right now):

In GH, there's five buttons that represent all notes that one can play. Actual melodies use more, and the meaning of the buttons changes based on the previous ones. Even with just five buttons, one can approximate a melody so that it mostly goes up and down as it should, but the actual intervals are incorrect. If I remember correctly, this actually corresponds to the first stages of the developmental continuum of singing (couldn't check because I don't have access to the full paper from where I'm writing). A video for refreshing one's memory:

In Heavy Rain, you use the right analog stick and shake the controller to approximate the movements of the characters. A video:

The differences between the two

• Heavy Rain, like most games with QTE:s, features some level of branching action based on whether the player reacts to the events on time, whereas in Guitar Hero, the song goes on, and the feedback from misses is implemented through scoring and sound effects. Also, if you make enough mistakes, the song ends prematurely and you get booed out (if I remember correctly - later music games usually let you always finish the songs)
• Guitar Hero and many other music games feature some form of a visual timeline, where one can see multiple approaching events and plan one's action. Action adventure QTE:s usually pop up one by one without such 'prediction horizon'.
• Action adventure QTE:s provide the player a reaction challenge, whereas rhythm games require coordination and precision.

I'm not that fond of QTE:s in fight scenes, because the reaction time is really difficult to tune - there seems to be only a tiny sweet zone between frustrating and boring, and it's different for every player. I'm more of a precision and coordination kind of guy, which brings me to wonder if one could actually have rhythm game style QTE timeline somehow overlaid in an action adventure fighting sequence. This would allow a more rapid pace of fighting, similar to many Hong Kong kung-fu movies:

An alternative to a timeline is to use multiple symbols that appear a predetermined time before the player should act, which most QTE:s do anyway, but in rhythm games the visual display of the remaining time to the correct moment is more prominent, as in Dance Evolution:

The benefit of the approach I'm proposing is that the timing and coordination challenges could be more interesting than the reaction challenge of traditional QTE:s, and only a single choreography without branches would be needed. The obvious problem is to design feedback that makes sense if the player's performance is less than perfect - do we want the "perfect", "good" etc. notifications from dance games or something else? Is the fight over if the player misses too many events? Should there be branching after all, and do we actually need two branches per event, one for being too late and the other for acting too early.

As a Google search will tell you, there's already many games with both rhythm and fighting elements. However, fighting to the beat of music as in this Kick Beat gameplay video or in Rhythm Fighter is a bit too much for me.

I really like the choreography, music and soundscape in Uncharted 3 and Heavy Rain. I just wish I could see more than one event in advance to get into the flow of the choreography and still have a reasonable challenge, similar to reading a note sheet when playing an instrument. This is what, e.g. Dance Central and Guitar Hero do, although due to the complexity of the player's realistic movements in DC, the choreography is not fully visualized on a timeline. Instead, one sees a timeline of flashcards, which have to be memorized in the training mode.

For Uncharted 3 and Heavy Rain, I think the action abstraction level is pretty much correct, but I guess I just feel strange trying to react quickly to button symbols shown on screen instead of the opponent's actual actions. The translation of symbols to actions feels more natural when given more time and when seeing multiple symbols at a time, maybe because the relations of the symbols to each other (e.g., the intervals between musical notes) help in decoding and executing the correct the motor sequence.

If anyone reads this and knows about this kind of experiments, please let me know.

Additional thoughts: The illusion of playing a guitar or fighting with QTE:s is not there at first, but at least it happens with me once I've become accustomed to the interface, I don't have to think of the real-world actions, I can focus on the results and feedback, and I get the timing right. Principles at work: We project ourselves to the avatar, our motor system assimilates the game interface through repetition, correlation is perceived as causation, our sensory integration system perceives two events as one if they happen close to each other in time. The same assimilation of the interface etc. happens of course with all action games, but the interesting thing is that it happens despite the player not having control or initiating the actions.

About dance game timeline displays and other choreography visualizations: The visualization of full-body movement over time in a compact and intuitive manner remains a research challenge, although various notations have been developed.

Trampoline games update

Uploaded a new video to YouTube and linked it to the work-in-progress manuscript submitted for CHI 2013. My previous post explains some of the goals and rationale behind the design.

Stanford AVP-STVP visit day 1

I'm part of the Aalto Ventures Program group visiting Stanford University this week. Aalto is a partner of Stanford Technology Ventures Program, and we try to learn how they support the exceptional startup ecosystem of the Silicon Valley. The not so modest goal of AVP is to apply STVP methods to make Helsinki the entrepreneurial hub of the nordic and baltic countries - Finland alone is maybe too small.

Here's an interesting nugget of information: in 2009, venture capital (VC) backed companies produced 11% of the jobs in the U.S. and their revenue exceeds 20% of the GDP (Source: National Venture Capital Association).

Other notes from the first day of the visit:

• STVP guiding principles (written on the walls): Every problem is an opportunity. The bigger the problem, the bigger the opportunity. Entrepreneurs do much more than imaginable with much less than seems possible.
• People here recommend Tina Seelig's new book inGenius: A Crash Course on Creativity. Haven't read it yet. Seelig's MOOC (Massively Open Online Course) on creativity has 39k participants. Should check that one too.
• ecorner.stanford.edu  has a lot of talks by company founders, broken down into a couple of minutes clips so that they are easier to use in teaching. "oh I want to have an example of scaling for the students, let's see". These were recommended to me, have to check out if/when I have time: Guy Kawasaki, Vinod Khosla, Jack Dorsey, Bill Gross, Carol Bartz, Marissa Mayer, Mark Pincus, Nick Earl, Erin Turner. Speakers are prepared to not only advertise their current company, but use it as a dataset using which to illustrate eternal stories. No TED talks, not too much polish. "What were you not told of beforehand?". Sometimes no slide decks, but just audience questions. Sometimes also important to protect young students from exploitation - not necessarily appopriate for a visiting speaker to invest in students.
• Steve Blank now has an online course at Udacity: How to Build a Startup. However, as Blank himself says, the online version lacks the feedback and rigor of the real Lean Launchpad course, which why Blank has also developed the Startup Weekend Next
• Must check ree.fi, REE stands for Roundtable in Entrepreneurship Education, and the latest REE conference was held recently in Finland. I hate that I didn't hear about it.
• Important to instill the entrepreneurial mindset from the start, and to provide practical examples of why to learn difficult things such as math. Couldn't agree more. Couldn't make myself motivated to learn much of the linear algebra or calculus that was taught during the first two years. I had no idea of where to apply it, although I've found every bit of what I managed to learn useful in my later computer vision and audio signal processing work.
• It might help Stanford students into the entrepreneurial mindset that on there's e.g., Google's first server and a replica of Hewlett's and Packard's garage on display. I don't know what would be the equivalent at Aalto. I just remember that there were old Nokia phones on display when I started my electrical engineering studies in the 90's. Should we put even more emphasis, e.g., in the game projects to produce high quality making-of materials so that we can accumulate and display the lessons learned for future students?
• Interesting: E&I Living Learning program equips dorms with maker spaces. Would definitely be useful at Otaniemi too.
• meetup.com is heavily used in Stanford and the Valley to organize groups to develop one's ideas and networks.
• Interesting form of conference: people pitch what they want to discuss (share, learn...), after which the space is divided into sections with topics. One may leave/move anytime to promote cross-pollination of ideas.
• Recruitment/interviewing in the Valley: Tell me about your top three blunders and what you learned from making the mistakes?

The joy of learning

Note to self: Watching my 11 months old daughter bubbling over with (drunk from the) joy of learning to stand without support ensures me that I'm on the right path when researching sports education technology and exergames. These are the experiences that I want to create and share. One might wonder if it's possible to experience something so strongly at an older age, but I must say that learning to backflip on a trampoline was pretty exhilarating a couple of weeks ago. 

About them piggies

Just tested Bad Piggies with my family. I have to say it's pretty good. 

Bad Piggies seems to be based on Amazing Alex, but adding two crucial elements: 1) a character to sympathesize with (the vocal reactions to the environment are priceless, similar to Porrasturvat / Stair Dismount) and 2) some realtime gameplay in "play mode". It's amazing how much the characters make me care compared to purely mechanical building games like Amazing Alex. Maybe it's just the mere exposure effect. Much like every parent of a three-year-old in Finland, I practically live in the Angry Birds transmedia world with the plush toys and playgrounds.

The bottom line is that getting three stars was never this satisfying in the birds empire - there's now clearly defined goals and the feedback has been amped up as well with manic partying to polka music. Having clear goals and feedback are preconditions of flow, and in hindsight it's easy to say that this is how it should've been from the start. In practice it's of course not easy to pull off. It's interesting to observe the progression from the first Angry Birds (just 1-3 stars, how to get all three left as guesswork), to Angry Birds Space (1-3 stars with a three-chord harmonic cadence so that having just two stars leaves you waiting for the resolution) and finally to Bad Piggies (a goal for each star, and the number of stars emphasized by both music and the facial expressions and body language of the piggy, ranging from "meh" to carnevalistic).

Just writing these thoughts down as a reminder of the smallest details making a huge difference.

Pitching and presenting games

Today we have the game project course kickoff at Media lab. Here's the handouts of a talk I prepared based on my own experiences, Samuli Syvähuoko's Game Developer Magazine article and Daniel Kahneman's book Thinking Fast and Slow. 

Here's also the links from the slides for your convenience:


TED talks about cognitive biases and manipulation, by excellent speakers
http://www.ted.com/talks/dan_ariely_asks_are_we_in_control_of_our_own_decisions.html
http://www.ted.com/talks/lang/en/rory_sutherland_life_lessons_from_an_ad_man.html

Lessig’s style of presenting
http://www.ted.com/talks/larry_lessig_says_the_law_is_strangling_creativity.html

Syvähuoko’s Game developer magazine article
http://gamedeveloper.texterity.com/gamedeveloper/fall2011cg#pg51

Steve Jobs presentation techniques
http://www.forbes.com/2009/10/08/steve-jobs-presentations-technology-meetings-09-tips_slide.html

Sell Your Ideas the Steve Jobs way:
https://www.youtube.com/watch?v=0q-wvAIeUgk

Steve Jobs demos the Machintosh
http://www.youtube.com/watch?v=G0FtgZNOD44

The original pitch of the Muppet Show
http://www.youtube.com/watch?v=9KorhvVQRUM


Kahneman’s book
http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374275637/ref=sr_1_1?ie=UTF8&qid=1348063450&sr=8-1&keywords=kahneman+thinking+fast+and+slow

Voice123, pretty good and affordable voiceovers.
http://voice123.com/

An article about one sentence summaries
http://www.rachellegardner.com/2010/06/the-one-sentence-summary/

Back at work

I just returned from 6 weeks of parental leave (isäkuukausi). Now working on a funding application for our trampoline training games and systems. Why? Because trampolines are cool and they practically give people superpowers, something I've tried to do earlier on screen in Kick Ass Kung-Fu and Kung-Fu Live.

Trampolines really seem to get people moving and the added jump height and reduced impact makes gymnastics training accessible to new target groups. The prices have dropped and there seems to be a trampoline in every backyard in the Finnish suburbs. Furthermore, a computer vision based trampoline system can improve safety by monitoring that there's only a single player bouncing at a time, something that indoor activity parks like Hoplop and Huimala require but don't have enough personnel to enforce. In this study, 74% of trampoline injuries were from situations where there was more than two people on the trampoline.

We aim to develop games and feedback tools to help in learning motor skills, and investigate the user experience, e.g., effect on skill acquisition and perceived competence. The importance of feedback in learning is well known, but the possibilities of novel technologies such as depth cameras (Kinect) are still relatively unexplored. So far, we have made a feasibility study, creating two simple Kinect games played on a trampoline. We have found that there's no fundamental obstacles for such systems - the safety net and elastic bed of the trampoline don't interfere too much with the tracking, the latency is not disturbing except for very low jumps (high jumping frequency), and it is possible for the user to watch a screen or screens while jumping.  Demo video:

In our research, we try to explore the following mechanisms: 

• Increasing body awareness with audiovisual feedback. Beginners often have no clue what their body is doing when they end up in unusual positions (hand stand, flipping in the air...). Displaying video and relevant measurements on a screen, either concurrently or with appropriate delay, can help in realizing what needs to be improved. The challenge is to keep the information visualization simple and focused and to find the best ways to control the system without interfering with training, e.g., using gestures or voice commands. See my previous post for examples of feedback systems.

• Manipulating the learning curve to help the user stay in the flow channel. For example, in gymnastics, the learning curve may actually be in the form of a staircase with high steps - you learn a skill, after which there may be little or no perceived progress until learning the next new skill, which may seem far. A good coach comes up with appropriate intermediate challenges and supporting exercises to keep the student motivated. Using a computer system, we can do the same but using quantitative measurements. For example, the first trampolining skill to learn is to jump high while staying in the middle of the trampoline. Using a Kinect, we can measure where the student lands and give a prize for executing 10 consecutive jumps that reach higher than 1 meter while staying inside a circle with radius of 50cm. As a more advanced example, after learning a single flip on a trampoline, a virtual on-screen obstacle can be added over which the student tries to do the flip higher and higher until reaching a height suitable for attempting a double flip.

• Manipulating perceived competence through the player's on-screen representation (avatar). The importance of perceived competence in staying motivated in sports training is well established in the literature. There's some recent studies showing that augmented feedback can improve perceived competence. We hypothesize that perceived competence can be increased by finding an optimal balance of developing real and virtual skills and attributes of the player. Some form of character development/upgrading is used in many games. In a physically interactive fighting game, the player may earn both real martial arts skills and abilities (kicking technique, flexibility, strength) and virtual ones (various armors to improve resistance to damage, improved jump height or kick strength to leap over or destroy obstacles, flaming hands that cause more damage, the ability to fly, posing to trigger spells...). Because the player identifies with the avatar, perceived competence is probably affected by the avatar's abilities. There's some anecdotal evidence to support the hypothesis, but we need to run comparative studies using different game versions and a standardized perceived competence questionnaire.

• Improving training safety by gamifying low risk exercises - in case of trampolining this would mean avoiding somersaults and flips to prevent neck injuries, and instead focus on basic jumping, landings (seat, front, back) and twisting around the vertical axis. The basic jumping games in the video above seem to motivate jumping by giving concrete goals and feedback, but we still need to come up with more examples and confirm the effects using suitable metrics, such as the Game Engagement Questionnaire.

Edit 4.9.: formatting, added the bullet about training safety