Sim2Real
A simulation-to-reality platform for autonomous driving research, from virtual agents to real robots.
My Ph.D. shifted from vehicle dynamics and optimal control toward learning-based motion prediction, but my interest in control engineering remained. Around the time I finished my Ph.D., I built a project at Linköping University that let me revisit all of it hands-on: a sim-to-real platform for autonomous driving (AD) research.
The goal was a platform where an entire AD software stack could be developed and validated in simulation, and then deployed onto real robotic vehicles while preserving consistent behavior.
I was interviewed about the platform by Linköping University. You can read the full article here:
Objective
The platform was built around a single guiding objective: develop a sim-to-real platform for autonomous driving research that supports a smooth transition from simulation to real-world deployment. Three requirements followed from it:
- The simulation had to model robot kinematics and sensor behavior with sufficient accuracy.
- The platform had to support realistic, rule-based traffic scenarios that translate to real-world conditions.
- The system had to include a comprehensive AD stack that runs consistently in simulation and in the real world.
Everything was built on ROS 2 and deployed using Docker, which kept the software environment identical across machines and across the sim-to-real boundary.
A modular framework
The software stack followed a modular framework: a set of submodules, each with a distinct responsibility, connected into a full perception, planning, and control pipeline. This was the standard design used in industry, and it was deliberate here. Modularity promotes interpretability, reusability, and maintainability, and a complete modular stack is an excellent starting point for targeted research into any single functionality.
The sections below walk through the stack from state estimation out to the low-level controllers that drove the wheels.
State estimation
In simulation, the full system state was available for free. On the real robots it had to be estimated from noisy sensors (onboard IMU, wheel odometry, and a camera-based positioning system) fused together.
Motion was described by a kinematic single-track (bicycle) model, which served as the prediction model for the estimator and as ground-truth dynamics in simulation:
\[\begin{aligned} \dot{x} &= v \cos\theta \\ \dot{y} &= v \sin\theta \\ \dot{\theta} &= \frac{v}{L}\tan\delta \\ \dot{v} &= a \end{aligned}\]Here \(x, y, \theta\) are the vehicle’s pose, \(v\) is velocity, \(a\) is acceleration, and \(\delta\) is the steering angle. The parameters were matched to the physical robot; for example, a wheelbase of \(L = 0.275\,\text{m}\). An Extended Kalman Filter (EKF) fused the sensor measurements against this model to produce the state estimates used during real-world experiments.
Route planning
The route planner solved the planning problem at the highest level: given a start and a goal, it computed the best path through the road network. The network was represented as a graph whose nodes were categorized (stop, yield, parking, regular); a mission, such as driving from one parking space to another, was defined over it, and the optimal plan was found with A* search. The resulting waypoints were handed down to the behavior layer.
Behavior layer
The behavior layer continuously adapted the robot’s actions to the current environment and mission. It was implemented as a Behavior Tree (BT), a modular, scalable structure that executed logical decisions such as checking for stops, handling yields, and managing intersections. It sent subtasks as waypoints down to the motion planner, communicated driving intent (go, stop, slow down) to the controller, and, because it was hierarchical, was easy to extend with new behaviors.
Motion planning
The motion planner was optimization-based. It found a time-optimal path near a set of reference waypoints \(\mathcal{W} = \{(w_{x_i}, w_{y_i})\}_{i=1}^{M}\) subject to the vehicle’s differential constraints \(f(\cdot)\):
\[\min_{u}\quad T + R\sum_{k=0}^{N-1}\lVert u_k \rVert^2 + Q_1 \sum_{k=0}^{N} g_{\text{path}}(x_k, s_k, \mathcal{W}) + Q_2 \lVert \sigma \rVert_1\] \[\begin{aligned} \text{s.t.}\quad & x_{k+1} = f(x_k, u_k), && \forall k \in \{0,\dots,N-1\}, \\ & s_{k+1} \geq s_k,\; s_0 = 0,\; s_N = 1, && \text{(path-progress constraints)} \\ & u_{\min} \leq u_k \leq u_{\max}, && \forall k, \\ & h(x_0, x_N, \sigma) \leq 0, && \text{(soft boundary constraints)} \\ & 0 \leq \sigma \leq \sigma_{\max}. && \text{(slack bounds)} \end{aligned}\]\(R, Q_1, Q_2\) are tunable weights and \(g_{\text{path}}\) penalizes deviation from the reference waypoints. Increasing \(Q_1\) pulled the optimized trajectory closer to the reference path, trading off against time-optimality.
Motion control
The motion controller executed the planned path by regulating steering and velocity. It used a decoupled architecture: lateral and longitudinal control were handled independently, which simplified tuning and reduced complexity.
- Lateral control used a Pure-Pursuit algorithm to compute the steering angle \(\delta\), with rate-limiting for smooth steering transitions.
- Longitudinal control used a cascaded structure: the Intelligent Driver Model (IDM) set the velocity (used directly in simulation), and a PI controller converted velocity commands into PWM signals for the real robots.
Demonstration
The platform ran a full AD stack (state estimation, route planning, motion planning, and control) in a simulated traffic scenario and then transitioned to real-world deployment on hardware. The video below shows the system in action, highlighting the consistency of system behavior across the sim-to-real boundary.
The technical write-ups and additional material are also on LinkedIn, and the project was featured in an interview by Linköping University:
Acknowledgements: this work was carried out at Linköping University, where it supported teaching and research in autonomous driving.