Abstract:
—
For my final project in both ICM and PComp, I developed body-positioning Kinect-controls for the Google Earth Flight simulator. The Kinect uses the z-position & x-position of the users head as well as their hips, to determine the pitch and roll of the aircraft; and it uses the average delta in y position of the left and right hands to determine speed. This allows people to flap and lean like a bird might and then fly around the planet!
Development:
—
The sketch that drives the whole application can be found here.
The sketch that drives the mouse movement from Arduino is below. There is some commented out text that controls a button input to start the game that has not been set up yet (as of this writing @ 9:40 am Dec. 5). But will be soon.
#include "Mouse.h" #include "Keyboard.h" String inData; int xpos; int ypos; int Speed; int StartPin = 8; // initialize button to begin game int greenLED = 9; int redLED = 10; boolean OnState = false; int seconds = 0; void setup() { Mouse.begin(); Keyboard.begin(); pinMode(StartPin, INPUT); // set Startpin to be input pinMode(redLED, OUTPUT); pinMode(greenLED, OUTPUT); digitalWrite(redLED, HIGH); digitalWrite(greenLED, LOW); Serial.begin(9600); // initialize serial communications Serial.setTimeout(100); // set the timeout for parseInt seconds = millis()/1000; } void loop() { if (Serial.available() > 0) { inData = Serial.readStringUntil(';'); // parseInt will read & output each full int until non-int character. xpos = Serial.parseInt(); ypos = Serial.parseInt(); Speed = Serial.parseInt(); if (1 > 0) { Serial.write('n'); // send it back out as raw binary data } } if (OnState == false){ int readStartPin = digitalRead(StartPin); if (readStartPin == HIGH) { digitalWrite(redLED, LOW); digitalWrite(greenLED, HIGH); Mouse.click(); OnState = true; } if (seconds > 30){ OnState = false; } } if (OnState == true){ if (Serial.available() > 0){ Mouse.move(xpos, ypos, 0); speedCheck(); }} } void speedCheck() { // use serial input to control velocity if (Speed > 3) { Keyboard.write('9'); if (Speed > 1 && Speed < 3) { Keyboard.write('9'); Keyboard.write('3'); } } if (Speed < 1) { Keyboard.write('3'); } }
Circuit & System Diagram
The circuit was very simple, just a line in from a button and some lines out to lights, and the system isn't very complex either. But there ended up being a fair bit of communication to clear up, as mentioned above. Once clean, though, it all seemed to simple to be true!
Fabrication:
—
The box I made is small - about 20”x20” - has two layers, and holds all my wires and circuit behind closed walls. I covered it in paper as it is a simple and clean solution, and it also was all made from salvaged materials from the junk shelf, but for the screws and nails, which I already had on hand in my stuff. For the final testing day, I had the pole for the button protruding straight up from the box, but I believe I should have it protrude straight away, about 1 meter, so that the user can stand at a distance that is equal to the minimum viewing distance from the kinect - and start the game. This will allow the game to be a bit more seamless of an intersection than I found it is today.
My final circuit turned to be a little more complex than I initially thought it may be - which is not so much a circuit by a unit in a system diagram. Even though now it’s only slightly more complicated than a single cable plugging the Arduino to the laptop. Basically, a big button is pressed by the user, lights switch from red to green, and the game begins. That is the extent of the tangible interaction. Then the user leans and flaps and controls the plane!
Reflection:
—
This project has been eye opening in many ways. From the concept, to the coding, to the friends anticipation, and their eventual experience of the work, every step of the way has been a journey, a process, a trial, and ultimately helpful.
Starting with the process, I’m quite happy I was able to push through and achieve what I intended to achieve with this project in allowing people to “fly.” It was many nights and hours fighting with the smallest things thinking I’d be better off simply pivoting, but I did not want to, and I’m glad I didn’t.
To the coding, it was tough. There were kinect connection issues, serial port issues, arduino issues… but at the end of the day I connected all the devices to provide real time feedback based on body positioning and limb actions. I have a hunch that children will catch on quicker than adults, I am excited and hopeful to test this hypothesis.
I did not build out objects or for loops in this code, and while I wish I had, I am ok that I didn’t, given that I had to figure out kinect and serial port issues, and write arduino code. There are many if statements and functions within functions, and the byte parsing communication was a doozy to figure out, though now that I know it, I can likely solve similar problems in about 3000% faster time. Looking back I can’t believe how long it took, but looking forward I can’t wait to how quickly projects like these will come together, and how much more able I will be to solve much more complex problems.
All along my friends have anticipated my completion, not out of some misplaced awe, but rather an delightful curiosity. Watching me flap my arms like a fool for the past several weeks, many have inquired as to what I’m up to with a smirk or outright smile. To be able to give them something close to what they imagine they might get - and are already excited about, is a delight.
And so too have their faces when they initially experience the simulator. Broad smiles, action, leaning, flapping, laughing - many great positive verbs. As of this writing I have not yet smoothed out the control of the mouse movements, but even if they are not smoothed out, the controller still works, and people love it.
