VR user affect estimation (current project)

Ongoing work. Currently I have taken the raw logs from the affect dataset collection project and post-processed them into useable features/labels. I have also implemented several machine learning models and trained them on the post-processed data. Going forward I will be focused on feature engineering and improving model architectures.

Private repository pending project completion

Details

Post-processing

Post-processing was done using a mixture of Pandas, NumPy, and custom functions
Signal logs were segmented to match stimuli presentations using timestamps
Segmented gaze signals were post-processed by:
- Performing blink detection using pupil size
- Out of range values were removed based on human physiology
- Pupil size outliers were removed based on their mean absolute deviation from that stimuli’s pupil size transition speeds
- Samples near temporal gap’s were removed as these gaps were caused by blinks, missing data, or transition speed outliers and could represent bad readings
- A low-pass filter was applied to the pupil data
- Fixations and saccades were determined using the normalized vectors representing eye direction
Movement information was extracted from raw device pose information
All signals were resampled to have identical sample frequencies and timestamps
Signals were normalized using min-max scaling

Models

Models are implemented using Tensorflow
Custom data loading classes were implemented using NumPy and Pandas to properly load, segment, and prepare data for models
Different types of models were implemented, including: support-vector machines, k-nearest neighbors, logistic regression, long short-term memory networks and 1-dimensional convolutional neural networks
- Sigmoid, RELU, and SoftMax activations were tested
- Different learning rates, epoch sizes, and model layers were tested
- Models are currently trained with leave one out validation (based on the domain)
- Confusion matrices, F1 scores, and accuracy are output during training