柠檬导航

News

Towards The Internet of Senses

The Internet of Senses (IoS) aims at providing comprehensive multisensory experiences that are nearly inseparable from reality and improve intelligent human-machine interaction.

This joint project between Aalto University and TII (Technology Innovation Institute), Abu Dhabi UAE, presents a human-machine interaction testbed for 6 DoF real-time immersion with haptic feedback towards realizing the Internet of Senses. In this project, we have worked on designing a real-life system that allows the remote control of a UAV for maximum immersion (6 Degrees of Freedom) while allowing optimal and reliable control of UAVs.
Internet of Senses

聽The testbed included:聽

 

 

Remote UAV

The remote UAV is equipped with an embedded computer, IoT sensors,  360鈼 camera and a 5G modem. 

VR users

Upon receiving the 360鈼 video stream, the user views the real-time stream through a Web platform and controls the UAV remotely using his body movements on the VR Treadmill platform.

Edge Server

The Edge server comprises the streaming module, control and monitoring module and the web server. 

Streaming module

This application is composed of an optimal that allows transmitting the real-time video stream to the web application with the lowest latency possible. It provides 360鈼 video to any device able to access the web. 

Web server

The web server serves the WebVR application. It is the interface to the user to view the information status of the UAV and 360鈼 video stream through an HTML5 video player adapted to play 360鈼 video. It manages the video stream from the streaming module and synchronizes different video inputs (UAV video streams) with the outputs (video players who are requesting a given stream). The 360鈼 video can be viewed by any device able to access a web browser. The choice of WebVR was mainly due to allowing an immersive view to any device that has access to a web browser starting from a simple card box to a HMD. 

Control and monitoring module

It is in charge of two functions: i) forwarding the user control commands from the web application to the flight controller module, and ii) updating the user about the censorial information of the UAV such as altitude, latitude, longitude and speed, as well as LTE and 5G-relevant information from the modem that is connected to the UAV. This information is integrated within the 360鈼 immersive view of the HMD. It is visualized by clicking on virtual elements within the immersive view

VR Treadmill

The treadmill is used to track the user movements (walking speed, heading, height) and translates them into drone maneuvers. We implemented an algorithm that gets the different sensors measurements from the treadmill platform and translates them into drone commands. We also exploited the drone鈥檚 sensors feedback to implement the so called haptic feedback. The implemented model takes into consideration the different drones鈥 movements and translates them into vibrating frequencies that are sensed by the VR User while flying the drone. Different maneuvers will lead to different torques at the copter motors level which would be translated into different vibration amplitudes at the treadmill level by the implemented algorithm. The implemented haptic feedback algorithm maps one to one the vibrations frequencies into different frequencies and amplitudes to give to the user different senses for different maneuvers.

A demo has been done at GITEX, the world鈥檚 largest tech event in Dubai. We had a user controlling a drone using his body movements based on the 4K 360 video feedback from Dubai while the drone was in Abudhabi which makes it around 100km with an end-to-end video delay <500ms.

We also tested the setting from Aalto University while the drone was in TII, Abudhabi UAE where a user would control the drone using VR joysticks and we had an end-to-end delay of 900ms.

For more information, please contact .

  • Updated:
  • Published:
Share
URL copied!

Read more news

Close-up of a complex scientific instrument with golden components and various wires, in a laboratory setting.
Press releases Published:

Time crystals could power future quantum computers

A time crystal, a long-life quantum system approaching perpetual motion, has been hooked up to its environment for the first time, unlocking an intriguing way to increase quantum computational and sensing power.
A person in black touches a large stone sculpture outside a brick building under a blue sky.
Campus, Research & Art, University Published:

Glitch artwork challenges to see art in a different light

Laura K枚n枚nen's sculpture was unveiled on 14 October at the Otaniemi campus.
Book cover of 'Nanoparticles Integrated Functional Textiles' edited by Md. Reazuddin Repon, Daiva Miku膷ioniene, and Aminoddin Haji.
Research & Art Published:

Nanoparticles in Functional Textiles

Dr. Md. Reazuddin Repon, Postdoctoral Researcher at the Textile Chemistry Group, Department of Bioproducts and Biosystems, Aalto University, has contributed as an editor to a newly published academic volume titled 鈥淣anoparticles Integrated Functional Textiles鈥.
Person standing outdoors in autumn, wearing a grey hoodie and green jacket. Trees in the background with orange leaves.
Appointments Published:

Introducing Qi Chen: Trustworthy AI requires algorithms that can handle unexpected situations

AI developers must focus on safer and fairer AI methods, as the trust and equality of societies are at stake, says new ELLIS Institute Finland principal investigator Qi Chen