Applications

Our Demo Applications

• Detecting static postures using an accelerometer.
• Detecting objects using a color sensor.
• Detecting frequency-based sounds using a microphone.
• Detecting temporal gestures using an accelerometer.

Other Possibilities

Accelerometer / Gyroscope / IMU

The BMI160 IMU has built-in step detection, no-motion detection, tap detection, and free-fall detection. See datasheet

Free Fall Detection

• http://cache.freescale.com/files/sensors/doc/app_note/AN3151.pdf

Fall Detection

• http://www.analog.com/library/analogdialogue/archives/43-07/fall_detector.html

Jump Detection

Activity Recognition

Gesture Recognition (using DTW), Wii-like

• Exemplar
• MAGIC

Microphone

Pipeline / feature considerations:

• SVM on raw FFT data can work well for single-pitch (or dominant frequency) sounds (e.g. bell, ringing a glass).
• FFTFeature provided by GRT is super noisy. It picks the highest magnitude frequencies, but with noise, the highest magnitude frequencies change randomly / frequently. Can we filter out background? Need a way to get dominant frequencies and their magnitudes together (parallel feature extraction and composition).
• Zero-crossing can also work well. PCA too.
• PCA is a CoreAlgorithm, it's not clear if we can use it as a feature extraction module.

Audio localization (directionality) using two (or more?) microphones.

Geometric sensing using audio:

Valkyrie Savage, Andrew Head, Björn Hartmann, Dan B. Goldman, Gautham Mysore, and Wilmot Li. 2015. Lamello: Passive Acoustic Sensing for Tangible Input Components. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1277-1280. DOI=http://dx.doi.org/10.1145/2702123.2702207

Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2161-2170. DOI=http://dx.doi.org/10.1145/2702123.2702414

Chris Harrison: scratch detection, touching phone w/ fingertip vs. knuckle

Frequency Detection

• http://www.instructables.com/id/Arduino-Frequency-Detection/
• http://www.instructables.com/id/Arduino-Pitch-Detection-Algorithm-AMDF/

Beat detection / spectrum analysis in music.

• http://dpeckett.com/beat-detection-on-the-arduino

Speech detection

• http://www.mirlab.org/conference_papers/International_Conference/Eurospeech%201997/pdf/tab/a0199.pdf)
• https://www.reddit.com/r/arduino/comments/23qk5r/electret_microphone_suitable_for_speech_detection/

Gender Identification

• http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1221721
• AUTOMATIC GENDER IDENTIFICATION OPTIMISED FOR LANGUAGE INDEPENDENCE
• http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=543213
• Automatic Gender Classification Using the Mel Frequency Cepstrum of Neutral and Whispered Speech: a Comparative Study - has a good summary of previous work.

Speaker Identification

Gunshot Detection

• http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5620932
• http://dl.acm.org/citation.cfm?id=2736143

Ring-Tone Detection

Irina Diaconita, Andreas Reinhardt, Delphine Christin, and Christoph Rensing. 2014. Bleep bleep!: determining smartphone locations by opportunistically recording notification sounds. In Proceedings of the 11th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MOBIQUITOUS '14). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 110-119. DOI=http://dx.doi.org/10.4108/icst.mobiquitous.2014.258035

Optical Sensing

Liquid recognition (from Interactive Device Design)

• http://sean-c.com/drinke.html

Or, more generally, recognizing objects using a color sensor

Pressure Sensing

Hand gesture recognition (Artem's paper)

Electrode / Capacitive Sensing

Grasp recognition: Brandon T. Taylor and V. Michael Bove, Jr.. 2009. Graspables: grasp-recognition as a user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 917-926. DOI=http://dx.doi.org/10.1145/1518701.1518842

Detecting who's sitting in a chair (fab class project) Detecting posture.

Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 483-492. DOI=http://dx.doi.org/10.1145/2207676.2207743