Keeping the lights on

Deep Q-learning is a subset of Q-learning, a type of machine learning that analyzes the results of inputs and seeks to maximize the reward for an action. In conventional Q-learning, different inputs from a user are mapped to output values in a table known as the Q-table. However, there are many challenges to creating a Q-table because it requires a large amount of computation as the number of input values increases. This can cause a computer to struggle and malfunction when the number of inputs and outputs reaches a certain size, leading Moradi to explore deep Q-learning. 

This diagram shows the difference between Q-learning with a Q-table and deep Q-learning.

Deep Q-learning varies from traditional Q-learning because it doesn’t need a Q-table, decreasing the capacity needed for computing processes. Instead, deep Q-learning uses a neural network, which is a type of machine learning model, to estimate outputs, and does not require the user to make a manual input for each output.

Moradi also chose deep Q-learning because of another factor important to power grid cybersecurity: It can be used for environments where parameters are unknown, just as the optimal attack and defense strategies aren’t known by a user before running the deep Q-learning simulation.

While deep Q-learning addresses the issue of computing power needed, the algorithm model the system uses to learn also needs to be optimized to ensure the best outcome. This is how Moradi picked the idea of modeling the situation as a stochastic game.

Cyber wargames

“I had a game theory class last semester, and I thought the attacker-defender relationship could be modeled as a game where each one tries to achieve its objectives,” Moradi says.

In a two-person stochastic game, two antagonistic parties play a game with multiple stages seeking to maximize their rewards. In this case, the reward for the attacker is to cause as much damage to the power grid as possible, while the reward for the defender is to use the best strategy to minimize damage to the power grid.

“One game move is like one iteration of attacking and defending; the attacker launches an attack, and the defender figures out a way to respond,” Lai explains. “In general, the defender is not able to protect all lines simultaneously because of limited resources. This highlights the need for Q-learning because the defender should wisely select the set of lines to protect.”

Powering up power grid cyberdefense

Ultimately, Moradi’s simulation isn’t used to defend against the grid in real time. Simulating cyberattacks and cyberdefense shows power grid weaknesses and the best ways for a human operator to defend against them to minimize cyberattack damage.

The research team says they are looking to further refine their work with future simulations, taking into account factors such as limited financial and human resources for defending the power grid.

Weng, whose research focuses on power systems, machine learning and cyberphysical systems, among other areas, says he anticipates that the research can be used for real-life cybersecurity.

“Being part of this project helped me appreciate the great idea of using this model to understand a physical system theoretically using AI solutions,” he says. “This will ensure that the work will transition to the real world, which is the core reason for this research.”