ÄûÃʵ¼º½

Supervisor: Prof. Tomasz Kucner (tomasz.kucner@aalto.fi)
Advisor: Dr. Farzeen Munir (farzeen.munir@aalto.fi)

Keywords: Autonomous vehicle, sensor fusion, Human Behavior modelling,  Deep Learning, Safe planning

Project Description 

Autonomous vehicles (AVs) must accurately predict pedestrian behavior to ensure safe and efficient navigation in urban environments, which presents challenges due to the variability of human movement and environmental factors. This thesis aims to develop a robust framework for pedestrian behaviour prediction using multimodal sensor data, including radar, LiDAR, and image data. The research will focus on building a complete data processing pipeline, implementing pseudo-labeling techniques for pedestrian behavior annotation, and designing machine learning models for behavior prediction. Advanced deep learning techniques such as Graph Neural Networks (GNNs), Transformer-based architectures like Social-STGCNN and PedFormer, and Recurrent Neural Networks (RNNs) with attention mechanisms will be explored. The study will benchmark these models against state-of-the-art approaches and analyze the impact of sensor fusion techniques on prediction accuracy. A systematic methodology will be followed, beginning with data preprocessing and synchronization, followed by model training and evaluation using performance metrics such as ADE, FDE, and classification accuracy.

Deliverable
The expected deliverables of this research include:
I.   Developed and tested pipeline for multimodal sensor data processing.
II.  A dataset with pseudo-labeled pedestrian behavior annotations.
III. Train a machine learning model for pedestrian intention and trajectory prediction.
IV. Performance benchmarking against existing state-of-the-art models.
V.  Research paper detailing the methodology, findings, and contributions.

Practical Information
- Python (high), Deep Learning (high), and Machine Learning
- PyTorch

References

• Bhattacharyya, Apratim, Daniel Olmeda Reino, Mario Fritz, and Bernt Schiele. "Euro-pvi: Pedestrian vehicle interactions in dense urban centers." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6408-6417. 2021.
• Salzmann, Tim, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. "Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data." In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pp. 683-700. Springer International Publishing, 2020.
• Liu, Yuejiang, Riccardo Cadei, Jonas Schweizer, Sherwin Bahmani, and Alexandre Alahi. "Towards robust and adaptive motion forecasting: A causal representation perspective." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17081-17092. 2022.
• Park, Yeong Sang, Jinyong Jeong, Youngsik Shin, and Ayoung Kim. "Radar dataset for robust localization and mapping in urban environment." In ICRA Workshop on Dataset Generation and Benchmarking of SLAM Algorithms for Robotics and VR/AR, Montreal. 2019.

Instructor: Maryam Kazemi Eskeri 
Group members: Gröhn Hilkka, Liu Xuanzhi, Pirttijärvi Viljami,Visa Alfred

Robotic systems often face challenges related to repeatability, especially in dynamic environments. For example, testing robotic behaviors in human-shared environments can be expensive, and potentially unsafe.  To tackle this problem, we can leverage Hardware in the Loop Simulation (HIL).

HIL simulation bridges the gap between purely virtual simulation and real-world testing. In this setup, robots are "realistic" while other components like obstacles, doors, or human movements are " simulated " as sensor reading. This method allows the robots to operate in real world accounting for their kinodynamic constraints, while receiving sensor inputs that mimic real-world conditions. 
Using these simulated data, we can inject specific types of data to tackle the problem of repeatability or problem with generating such data in a lab setup.  

Our goal is to develop a robust HIL system for the robotics laboratory. The project involves publishing simulated data on the Robot Operating System (ROS). Each Raspberry Pi, connected to a projector, subscribes to the data and projects the corresponding section of the dynamic environment onto the floor. This approach eliminates the need for physical calibration of the projectors, enabling seamless integration of the simulated environment with the real world. 

This course on Mobile Robotics provides an in-depth exploration of key concepts and advanced techniques essential for the development of autonomous mobile robots. Students will learn about robotic locomotion, sensing and perception, and the probabilistic approaches to mapping and localization. The curriculum covers contemporary software tools, simultaneous localization and mapping (SLAM), inertial navigation systems (INS), and global navigation satellite systems (GNSS). Additionally, it delves into task and motion planning, as well as control systems that ensure precise and efficient robot operation. By the end of the course, students will be proficient in designing, implementing, and evaluating sophisticated mobile robotic systems, preparing them for cutting-edge research and careers in robotics.

LEARNING OUTCOMES

Upon completion of this course, the student will be able to design a comprehensive, high-level architecture for a mobile robotic system capable of addressing diverse challenges in the domain of field, indoor, service, agricultural robotics and similar, as well as address some challenges in the field of and autonomous vehicles.

This proficiency will encompass the ability to:

• Identify and name the primary challenges encountered by mobile robots in their respective fields and propose effective solutions.
• Illustrate the composition of a robotic system well-suited to the specific problem, involving the selection of appropriate subsystem instances. The student will be able to present the interplay between these subsystems, utilizing accurate terminology.
• Assess, select, and, to a limited extent, apply and implement fundamental methodologies and algorithms relevant to the identified challenges.

Study Material 

• Alonzo Kelly, CMU, Mobile Robotics: Mathematics Models and Methods, Cambridge University Press, 2014;
• Trun & al, Probabilistic robotics, MIT Press 2005;
• Siegwart, Nourbakhsh, Introduction to Autonomous Mobile Robots, MIT Press (2nd ed.)

This course is designed to provide students with hands-on, practical experience through collaborative project work. It emphasizes teamwork, project management, and real-world application of engineering concepts, preparing students for professional challenges in the field.

Course Structure:

• The core of this course involves group projects consisting of 4-5 students. These projects foster teamwork and hands-on application of theoretical knowledge.
• All topics are proposed by the EEA department, ensuring relevance and expert guidance. Students cannot propose their own topics.
• Each project is supported by an expert instructor from the department, providing valuable insights and guidance throughout the course.
• Before the course kick-off lecture, students will find a list of project topics in the "Project topics" section. Each student should select five topics based on their preferences.
• The teachers' board will form groups after an overall assessment, taking into account student preferences and available positions.
• Within the first 2-3 weeks, each group will select a project manager from among themselves.
• Midway through the course, there will be a business case exercise related to your project topic, supplemented with relevant lectures.

• The Final Gala marks the completion of all technical work on the projects. Following this, students will have about one week to complete their final report.

  
  Assessment:

  The course grade is a composite of self-assessment, peer-evaluation within the project group, and teacher assessment. These evaluations occur twice: once after week 12 and again after week 21. Each student will receive a personal grade ranging from 1 to 5.

  
  This course not only aims to enhance your technical skills but also to develop your ability to work effectively in teams, manage projects, and apply business insights to technical problems. Join us for an engaging and enriching learning experience!