Research&Innovation Actions

Title of Proposal

Multisensory-enhanced Innovative Learning Network

(MultILearNet)

 

List of participants

 

Participant No *

Participant organisation name

Country

1 (Coordinator)

Friedrich-Alexander-Universität
Erlangen-Nürnberg

Germany

2

National Technical University of Ukraine “Kyiv Polytechnic Institute”

Ukraine

3

Athlone Institute of Technology

Ireland

4

Holografika

Hungary

5

Imaginary

Italy

6

LiveU

Israel

7

FORTH

Greece

8

Brunel University

UK

9

Dublin City University

Ireland

10

RGB Screen Networks Ltd.

Ireland

11

Slovak University of Technology

Slovakia

12

Middle East Technical University

Turkey

 

1.      Excellence

The MultILearNet proposal will address the need for robust and fit-for-purpose digital technologies for learning. The aim is to create multisensory learning environments that represent realistically the natural environment and evaluate its technical and pedagogical effectiveness. The consortium combines industry and academia cooperation to develop an innovative approach to learning by using multisensory environment to mimic the real life environment and create adaptive learning, use augmented cognition technologies, virtual environments/virtual worlds to real-life learning situations.

Traditional learning was based on students in a passive mode with the tutor or teacher presenting information. This has currently evolved to support various blended learning approaches. The next logical step is the creation of multisensorial multimodal media learning environments and this is the key aim of the MultiLearNet project.

The importance of using naturalistic environments to study human perception and information systems has been emphasized by many researchers (Findlay & Gilchrist, 2003; Hayhoe & Ballard, 2005; Land, 2003). Naturalistic environments contain complex sensory patterns, and our perceptual systems evolved and developed to allow us to sense and interpret these patterns. Until recently, it has been difficult for scientists to conduct carefully controlled experiments with naturalistic stimuli. However, new technologies, such as advanced computer graphics chips and virtual reality environments, are allowing researchers to investigate human perception in more realistic settings than has been possible in the past.

Multimedia systems try to take advantage of human senses to facilitate human communication and improve educational delivery and learning. Multimedia systems have the potential to make more efficient use of the human perceptual and cognitive capabilities because the human brain has evolved in a multisensory environment. Therefore the use of multiple media can make the interaction and learning more natural. However, existing multimedia systems address mostly only two senses by using two communication channels (visual and audio) of the five human senses (sight, hearing, taste, smell, and touch), limiting its potential efficiency. Though the current progress in technology enables to register and reproduce more types of information. Examples of new types of devices are electronic nose, electronic skin, etc. The task of multimodal data representation, compression, security, transfer, storage, retrieval, and reproducing is topical and its solving creates new amazing opportunities for learning / teaching, entertainment and telecommunications. The range of potential applications of MultILearNet is quite wide. One of the most exiting areas of application lies within education and training where MultILearNet technology will be able to recreate real like environments and especially environments not easily accessible by the majority of people like, volcanos, the artic, mountains, deep see etc.  More precisely, educational topics that can be addressed are: geography, environmental education, physics etc. Hardware necessary for such applications include 3D scanners, 3D projectors, surround sound systems, Olfactory devices and high-speed network connections which are becoming wider available and its costs decreases considerably. To strengthen the impression of a presence a mechanism for tactual information registration and reproduction is necessary.

 

The use of smell detectors and reproducers is also necessary, since the environment can be characterized by specific smell (e.g. sea, forest, etc.) as well as the interlocutor can also be a bearer of specific smell – e.g. he/she can use some perfume. Thus, information about the real object or person and its or his/her environment consists of visual information, audio information, tactile information, information about smell and environmental conditions (temperature, humidity, pressure).

 

MultILearNet will achieve its goal by creating 4 such installations where real environments will be reconstructed using either data obtained by the users or data freely available to the public through research institutes. The installations will be created in different partner locations and in different types of places to achieve accessibility and sustainability. One installation will feature the highest form of all considered peripheries, including a full-scale light-field 3D display for multi-user glasses-free visualization, while the other installations will use lower cost alternatives (such as desktop 3D displays, 3DTVs or head-mounted displays) to maintain an acceptable level of overall project cost.

 

One installation will be developed in Ireland at Cinema environment at RGB Screen Networks in Ireland. RGB Screen Networks currently have a commercially operating multisensory theatre that can support up to nine simultaneous users. It can provide effects such as 2D & 3D video, audio, a moveable platform (e.g. for earthquake simulation), as well as multiple wind, mist, snow and smoke devices in addition to strobe lighting effects. Such installations will guarantee two things, first, accessibility due to the number of people using the cinemas (their current estimated annual footfall is approx. 25000 per annum) and second sustainability of the project developments because RGB will sustain the installation well beyond the end of the project. The second installation will be developed at the technological museum in Kiev which will also secure access to a large number of visitors and users and its wide range of pedagogical activities. The third installation will be created at the media lab of ILI where can be used to test the effectiveness of multisensory learning beyond the duration of the project. Finally, the place of the last installation will be decided during the project based on the requirements analysis and contact with museums to achieve as wider accessibility and success as possible. One such possible candidate will be the Deutsches Museum in Munich which is one of the largest in Europe and very active in educational programmes. Alternatively, MultILearNet will use Oculus Rift-like head mount device technology to create more cost effective and adaptive environment without losing in the other main sensory modes.

 

The programme will validate the technology and the educational benefits of the multisensory learning approach in real-life pilots in the consortium countries were the installations will be based and in the rest of the countries the head mount devices will be tested with at least 20-30 users evaluating the technical and pedagogical aspects of the devices. Additionally the sustainability of the installations beyond the end of the project will be guaranteed because some of those installations will be created by private companies and museums which in turn will sustain them since they will be part of their normal businesses.

 

One of the tasks to be solved in the proposed project consists in the development of new data formats for optimal representation of complex data about a real object. The list of issues related to this project:

 

1. Complex data representation, including data compression and data security

2. Registration and reproducing of 3D images

3. Registration and reproducing of 3D sound

4. Registration and reproducing of olfactory data (smells)

5. Registration and reproducing of touches

6. Registration and reproducing of environmental conditions

7. Multisensory learning

 

In order for MultILearNet to realise its goals the project sets the following objectives presented in the section below.