The sound of silence: new video tech looks beyond the internet of things

The use of advanced video technology to listen to objects and buildings has important implications for engineering, construction and med-tech

New video technology could help push the boundaries of space exploration. Photograph: NASA Archive/Alamy

Recent discussions about video surveillance relate to online privacy and how technology such as wearable devices and CCTV can compromise people’s privacy. This has led to the development of more technology to counter these intrusions.
Anti-tracking and anti-surveillance software allows users to control the visibility of their online activities. Wearable devices such as Google Glass and the Apple Watch and Facebook’s tag suggest feature, which enables people to be tagged in photos and video footage without their consent, have led to concerns about the right to anonymity when engaging in activities offline and the creation of makeup, clothing and accessories that are designed to block facial recognition.
Another hot topic is the internet of things (IoT), whereby connected devices collect and communicate user data. This too is creating concerns about privacy and anonymity. But recent research using high-speed video technology reveals that there are a lot more to objects than user data. In fact, there are more to objects than meets the eye.

Transforming objects into visual microphones

Computer vision expert, musician and MIT PhD student Abe Davis has developed video technology that reveals an object’s hidden properties. One of those properties is sound.

Davis uses high-speed silent video to capture and reproduce sound, including music and intelligible speech, from the vibrations it makes on inanimate objects. His starting point was software developed by Michael Rubinstein and others at MIT. This software amplifies subtle motions in video so that they become large enough to see, for example, revealing the pulse in someone’s wrist in sufficient detail to measure their heart rate. Davis decided to apply the concept of extending our sense of touch to video by developing software that effectively uses video to “hear”.

Sound causes all objects to vibrate, but the vibrations are usually too subtle and too fast to be visible. Davis records these vibrations on high-speed video and analyses them in a way that extracts the sounds that created them, thereby turning objects into visual microphones. Initially, he recorded music from the vibrations it made on an empty crisp packet in the room where it was being played. He then extended the capability to recording conversations behind soundproof glass from a distance and capturing music by filming the vibrations of plastic earphones attached to a laptop computer.

Davis’ technique is non-invasive and does not require specialist camera equipment, although higher-quality equipment captures more detail.
This is not groundbreaking in terms of surveillance, as laser microphones have long been used to record conversations from a distance although it can achieve similar results without sophisticated and expensive equipment – using an ordinary DSLR camera and (potentially open-source) software. This is about pushing the boundaries of science and technology to help us understand objects in new ways. Davis describes it as “a new way to capture our surroundings”.

Using sound to identify structural defects

Davis developed this technique further, using sound vibrations to apply simulated forces to real-world objects. In his TED talk, he uses five seconds of high-speed video to create a 3D fully animated object and test its reactions. “This is a powerful way to look at the world because it lets us predict how objects will respond to new situations,” he explains. 

Davis’ latest research project builds on established theory that connects the physical properties of objects to their vibrations. He sees the most immediate uses of the technology in specialised engineering applications, which involve analysing the properties of different structures. He recently co-authored a paper about the use of tiny vibrations in video to identify material properties such as density and stiffness in fibreglass, wood and metal rods and fabrics. This could prove particularly useful in the field of non-destructive testing, or determining materials’ physical properties – and weaknesses – without damaging them. For example, it might be possible to identify structural defects in an airplane’s wing by analysing video of its vibration during flight.
This technique also has potential applications in civil engineering through capturing invisible frequencies to find and repair weaknesses in structures and buildings. “An exciting possibility for earthquake-prone areas is the potential to detect when there are problems in a structure,” says Davis. “Maybe it’s safe when you build it, but what about 10 or 20 years later? Our work could detect some of these problems without taking the structure apart. This could also be useful when looking at maintaining historical structures. Where do they need reinforcement and how much do they need? There are a lot of times when we want to keep something from collapsing while preserving as much of its original character as possible. Our technology could help engineers figure out how to do that.”

Davis envisages applying this methodology to specialised engineering tasks, and eventually developing it into a new type of imaging that is integrated into a broader range of fields and applications.
Another immediate possibility is the creation of hyper-real special effects for movies and games – and for use in architectural design.

Pushing the boundaries of medicine and science

Davis’ non-destructive, non-invasive techniques present interesting possibilities in the med-tech space. It offers potential benefits in the same way as Rubinstein’s software’s ability to “see” a heartbeat and physical changes related to blood flow. A comment posted on Davis’ TED talk suggested the possibility of developing a biomarker for Parkinson’s disease by capturing rapid eye movement and reactions to visual/auditory stimuli. Other comments suggest it could be used to improve the accuracy of remote diagnosis – ie a sophisticated Skype doctor. 

It could even push the boundaries of exploration, for example, to capture sound in space – or remotely discover the properties of other planets. “Space is another interesting direction,” Davis observes. “It’s like a giant soundproof glass window – light gets through but sound doesn’t. This technology could give us a new way to ‘listen’ to the cosmos.”

Other comments introduce some fun suggestions that this could reveal the science behind the theory that talking to plants encourages them to grow – it might be that they are actually listening.
Source : theguardian

We Are Fossasia Stay Connected With Us On Twitter . . . ! ! !