A Muse EEG headband, my phone camera and a set of models running on my own PC, driving a digital human that sees, listens and answers aloud. The headband data never leaves the machine.
The phone camera as the input, producing the same output as the clip above. My face, head, hands and fingers move the same MetaHuman in Unreal, over the same Live Link stream, through the same mixer. Only the sensor has changed, and with it what can be picked up: nothing is on my head here, and the camera sees what a headband never could, which is where my hands are and what each finger is doing.
The camera used as vision rather than as control. Nothing in this clip is driving a face. The camera is pointed at objects on the desk and the avatar answers spoken and typed questions about what it can see: what the thing is, how to cook it, how hot it is. Each answer starts from a still taken at that moment, read by on-device text recognition and a local captioning model, written by a local language model and spoken aloud. The character is different here, the VRoid avatar in the browser rather than the MetaHuman in Unreal, because nothing is driving a face and Unreal does not need to be open.
A variation on the third clip rather than a fourth idea. The same pipeline, pointed at a stored photograph with a reference sheet behind it, which is how the accuracy of the answers gets checked rather than taken on trust. Same avatar as the clip above.
A question takes a fresh still from the phone at that moment. Windows reads any text in it, a small local captioning model describes the scene, and a local language model writes the answer from both. The avatar speaks it in a voice cloned from a recording of my own, with mouth timing generated for each sentence as it is produced, so the first sentence plays while the next is still being made. Measured lip sync drift is single digit milliseconds, and typical time from question to the first spoken word is a little over two seconds.