Behdad Chalaki and Andreas A. Malikopoulos

Deploying optimal control strategies for coordination of connected and automated vehicles (CAVs) often requires re-evaluating the strategies in order to respond to unexpected changes in the presence of disturbances and uncertainties. In this paper, we first extend a decentralized framework that we developed earlier for coordination of CAVs at a signal-free intersection to incorporate replanning. Then, we further enhance the framework by introducing a priority-aware resequencing mechanism which designates the order of decision making of CAVs based on theory from the job-shop scheduling problem. Our enhanced framework relaxes the first-come-first-serve decision order which has been used extensively in these problems. We illustrate the effectiveness of our proposed approach through several numerical simulations.

Behdad Chalaki and Andreas A. Malikopoulos

In this paper, we provide a hierarchical coordination framework for connected and automated vehicles (CAVs) at two adjacent intersections. This framework consists of an upper-level scheduling problem and a low-level optimal control problem. By partitioning the area around two adjacent intersections into different zones, we formulate a scheduling problem for each individual CAV aimed at minimizing its total travel time. For each CAV, the solution of the upper-level problem designates the arrival times at each zones on its path which becomes the inputs of the low-level problem. The solution of the low-level problem yields the optimal control input (acceleration/deceleration) of each CAV to exit the intersections at the time specified in the upper-level scheduling problem. We validate the performance of our proposed hierarchical framework through extensive numerical simulations and comparison with signalized intersections, centralized scheduling, and FIFO queuing policy.

Behdad Chalaki and Andreas A. Malikopoulos

In this article, we establish a decentralized optimal control framework for connected and automated vehicles (CAVs) crossing multiple adjacent, multilane signal-free intersections to minimize energy consumption and improve traffic throughput. Our framework consists of two layers of planning. In the upper level planning, each CAV computes its optimal arrival time at each intersection recursively along with the optimal lane to improve the traffic throughput. In the low-level planning, we formulate an energy-optimal control problem with interior-point constraints, the solution of which yields the optimal control input (acceleration/deceleration) of each CAV to cross the intersections at the time specified by the upper level planning. Moreover, we extend the results of the proposed bi-level framework to include a bounded steady-state error in tracking the optimal position of the CAVs. Finally, we demonstrate the effectiveness of the proposed framework through simulation for symmetric and asymmetric intersections and comparison with traditional signalized intersections.

Behdad Chalaki and Andreas A. Malikopoulos

In this paper, we provide a decentralized optimal control framework for coordinating connected and automated vehicles (CAVs) in two interconnected intersections. We formulate a control problem and provide a solution that can be implemented in real time. The solution yields the optimal acceleration/deceleration of each CAV under the safety constraint at “conflict zones,” where there is a chance of potential collision. Our objective is to minimize travel time for each CAV. If no such solution exists, then each CAV solves an energy-optimal control problem. We evaluate the effectiveness of the efficiency of the proposed framework through simulation.

Behdad Chalaki, Logan E. Beaver, A M Ishtiaque Mahbub, Heeseung Bang and Andreas A. Malikopoulos.

Logan E. Beaver, Behdad Chalaki, A M Ishtiaque Mahbub, Liuhui Zhao, Raymond Zayas and Andreas A. Malikopoulos.

The implementation of connected and automated vehicle (CAV) technologies enables a novel computational framework to deliver real-time control actions that optimise travel time, energy, and safety. Hardware is an integral part of any practical implementation of CAVs, and as such, it should be incorporated in any validation method. However, high costs associated with full scale, field testing of CAVs have proven to be a significant barrier. In this paper, we present the implementation of a decentralised control framework, which was developed previously, in a scaled-city using robotic CAVs, and discuss the implications of CAVs on travel time.

Behdad Chalaki, Logan E. Beaver and Andreas A. Malikopoulos.

Roundabouts in conjunction with other traffic scenarios, e.g., intersections, merging roadways, speed reduction zones, can induce congestion in a transportation network due to driver responses to various disturbances. Research efforts have shown that smoothing traffic flow and eliminating stop-and-go driving can both improve fuel efficiency of the vehicles and the throughput of a roundabout. In this paper, we validate an optimal control framework developed earlier in a multi-lane roundabout scenario using the University of Delaware’s scaled smart city (UDSSC). We first provide conditions where the solution is optimal. Then, we demonstrate the feasibility of the solution using experiments at UDSSC, and show that the optimal solution completely eliminates stop-and-go driving while preserving safety.

Behdad Chalaki, Logan E. Beaver, Ben Remer, Kathy Jang, Eugene Vinitsky, Alexandre M. Bayen and Andreas A. Malikopoulos.

In this article, we demonstrate a zero-shot transfer of an autonomous driving policy from simulation to University of Delaware’s scaled smart city with adversarial multi-agent reinforcement learning, in which an adversary attempts to decrease the net reward by perturbing both the inputs and outputs of the autonomous vehicles during training. We train the autonomous vehicles to coordinate with each other while crossing a roundabout in the presence of an adversary in simulation. The adversarial policy successfully reproduces the simulated behavior and incidentally outperforms, in terms of travel time, both a human-driving baseline and adversary-free trained policies. Finally, we demonstrate that the addition of adversarial training considerably improves the performance of the policies after transfer to the real world compared to Gaussian noise injection.

Kathy Jang, Eugene Vinitsky, Behdad Chalaki, Ben Remer, Logan Beaver, Andreas A. Malikopoulos, and Alexandre Bayen.

Using deep reinforcement learning, we successfully train a set of two autonomous vehicles to lead a fleet of vehicles onto a round-about and then transfer this policy from simulation to a scaled city without fine-tuning. We use Flow, a library for deep reinforcement learning in microsimulators, to train two policies, (1) a policy with noise injected into the state and action space and (2) a policy without any injected noise. In simulation, the autonomous vehicles learn an emergent metering behavior for both policies which allows smooth merging. We then directly transfer this policy without any tuning to the University of Delaware’s Scaled Smart City (UDSSC), a 1:25 scale testbed for connected and automated vehicles. We characterize the performance of the transferred policy based on how thoroughly the ramp metering behavior is captured in UDSSC. We show that the noise-free policy results in severe slowdowns and only, occasionally, it exhibits acceptable metering behavior. On the other hand, the noise-injected policy consistently performs an acceptable metering behavior, implying that the noise eventually aids with the zero-shot policy transfer. Finally, the transferred, noise-injected policy leads to a 5% reduction of average travel time and a reduction of 22% in maximum travel time in the UDSSC.

Behdad Chalaki and Andreas A. Malikopoulos.

In this paper, we extend a framework that we developed earlier for coordination of connected and automated vehicles (CAVs) at a signal-free intersection to incorporate uncertainty. Using the possibly noisy observations of actual time trajectories and leveraging Gaussian process regression, we learn the bounded confidence intervals for deviations from the nominal trajectories of CAVs online. Incorporating these confidence intervals, we reformulate the trajectory planning as a robust coordination problem, the solution of which guarantees that constraints in the system are satisfied in the presence of bounded deviations from the nominal trajectories. We demonstrate the effectiveness of our extended framework through a numerical simulation.

Nakka Sumanth, Behdad Chalaki and Andreas A. Malikopoulos.

The steady increase in the number of vehicles operating on the highways continues to exacerbate congestion, accidents, energy consumption, and greenhouse gas emissions. Emerging mobility systems, e.g., connected and automated vehicles (CAVs), have the potential to directly address these issues and improve transportation network efficiency and safety. In this paper, we consider a highway merging scenario and propose a framework for coordinating CAVs such that stop-and-go driving is eliminated. We use a decentralized form of the actor-critic approach to deep reinforcement learning-multi-agent deep deterministic policy gradient. We demonstrate the coordination of CAVs through numerical simulations and show that a smooth traffic flow is achieved by eliminating stop-and-go driving. Videos and plots of the simulation results can be found at this supplemental

Behdad Chalaki and Andreas A. Malikopoulos

Connected and automated vehicles (CAVs) can alleviate traffic congestion, air pollution, and improve safety. In this paper, we provide a decentralized coordination framework for CAVs at a signal-free intersection to minimize travel time and improve fuel efficiency. We employ a simple yet powerful reinforcement learning approach, an off-policy temporal difference learning called Q-learning, enhanced with a coordination mechanism to address this problem. Then, we integrate a first-in-first-out queuing policy to improve the performance of our system. We demonstrate the efficacy of our proposed approach through simulation and comparison with the classical optimal control method based on Pontryagin’s minimum principle.

Meera Ratnagiri, Clare O’Dwyer, Logan Beaver, Logan Beaver, Heeseung Bang, Behdad Chalaki and Andreas A. Malikopoulos.

In this paper, we investigate the problem of a last-mile delivery service that selects up to N available vehicles to deliver M packages from a centralized depot to M delivery locations. The objective of the last-mile delivery service is to jointly maximize customer satisfaction (minimize delivery time) and minimize operating cost (minimize total travel time) by selecting the optimal number of vehicles to perform the deliveries. We model this as an assignment (vehicles to packages) and path planning (determining the delivery order and route) problem, which is equivalent to the NP-hard multiple traveling salesperson problem. We propose a scalable heuristic algorithm, which sacrifices some optimality to achieve a reasonable computational cost for a high number of packages. The algorithm combines hierarchical clustering with a greedy search. To validate our approach, we compare the results of our simulation to experiments in a 1:25 scale robotic testbed for future mobility systems.

Raymond M Zayas, Logan Beaver, Behdad Chalaki, Heeseung Bang and Andreas A. Malikopoulos

The increasing demand for emerging mobility systems with connected and automated vehicles has imposed the necessity for quality testing environments to support their development. In this paper, we introduce a Unity-based virtual simulation environment for emerging mobility systems, called the Information and Decision Science Lab’s Scaled Smart Digital City (IDS 3D City), intended to operate alongside its physical peer and its established control framework. By utilizing the Robot Operation System, AirSim, and Unity, we constructed a simulation environment capable of iteratively designing experiments significantly faster than it is possible in a physical testbed. This environment provides an intermediate step to validate the effectiveness of our control algorithms prior to their implementation in the physical testbed. The IDS 3D City also enables us to demonstrate that our control algorithms work independently of the underlying vehicle dynamics, as the vehicle dynamics introduced by AirSim operate at a different scale than our scaled smart city. Finally, we demonstrate the behavior of our digital environment by performing an experiment in both the virtual and physical environments and compare their outputs.