Evaluation

Evaluation Module

The Evaluation module is used for real-time evaluation of the scenarios in which vehicles are located. It includes five evaluation dimensions, and the evaluation results can be used for decision-making or scenario extraction, providing data for subsequent testing.

Now let's introduce these five dimensions one by one.

Consumption

We apply the following formula to calculate the energy consumption of a vehicle:

\[ P = (v * a)^2 + v^3 \]

Here, \(v\) and \(a\) represent the vehicle's speed and acceleration, respectively.

Under free driving (cruising) conditions, the vehicle's speed is relatively high, and the acceleration is relatively small, resulting in higher overall energy consumption, with speed being the dominant factor. In cases where the vehicle needs to brake urgently, the speed decreases sharply, and the acceleration is relatively high, resulting in lower overall energy consumption but with discontinuous changes.

Collision

We use the two-dimensional Time-to-Collision (TTC) to calculate possible collisions. The two-dimensional TTC is defined as the time it takes for two vehicles to collide if they continue to travel in their current speed and direction. TTC indicates potential collision risks and can be used to evaluate the safety of trajectory planning algorithms.

Here, we have referred to the implementation of nuplan and simplified it according to our data format.

Orientation

We use the deviation of the vehicle from the centerline of the lane (the angle between the velocity direction and the lane centerline) as the orientation.

\[ (\sin(\theta_{ego}) - \sin(\theta_{ref}))^2 \]

Here, \(\theta_{ego}\) and \(\theta_{ref}\) represent the angles of the ego vehicle and the projection point of the ego vehicle on the reference line (in the world coordinate system). There are three points to note here: 1. For the sake of presentation, we use \(\sin\) for normalization. 2. The meaning of the deviation angle is not significant within the junction, so \(\theta_{ref}\) here actually refers to the "nearest point on the NormalLane in the historical trajectory". 3. Also for the sake of presentation, we have added a delay term to make the turning display smoother.

Orientation can reflect the turning behavior of the vehicle, especially in scenarios that require unprotected left turns, which can be used to filter out some scenarios for testing based on the Orientation results.

Discomfort

We use the square of jerk to represent discomfort.

\[ \left(\frac{da}{dt}\right)^2 \]

Here, \(a\) represents the acceleration. Jerk can show the impact of sudden braking and sudden acceleration on passengers during vehicle motion.

Offset

We use the square of the lateral coordinate \(d\) of the vehicle from the centerline of the lane (within the junction, this dimension is not calculated) as the offset.

\[ d^2 \]

Here, \(d\) represents the lateral coordinate of the vehicle in the Frenet coordinate system.

Offset can be used to detect lane-changing behavior of vehicles. Since LimSim's default planner is based on the Frenet coordinate system, this dimension can be conveniently calculated.

Visualization

We use radar charts (also known as star charts) to visualize the results of the five evaluation dimensions. To ensure consistency in the visualization, we apply appropriate transformations to the results.

Data Retrieval

We store the real-time evaluation results in the database for scenario extraction and analysis. We can use the following code to convert the evaluation results into a DataFrame:

from utils.load_data import load_data_from_database

evaluation_df = load_data_from_data_base(path_to_data_base, 'evaluation')