Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading

Amarsagar Reddy Ramapuram Matavalam, Kishan Prudhvi Guddanti, Yang Weng, Venkataramana Ajjarapu

Research output: Contribution to journalArticlepeer-review


This paper describes how domain knowledge of power system operators can be integrated into reinforcement learning (RL) frameworks to effectively learn agents that control the grid&#x0027;s topology to prevent thermal cascading. Typical RL-based topology controllers fail to perform well due to the large search/optimization space. Here, we propose an actor-critic-based agent to address the problem&#x0027;s combinatorial nature and train the agent using the RL environment developed by RTE, the French TSO. To address the challenge of the large optimization space, a curriculum-based approach with reward tuning is incorporated into the training procedure by modifying the environment using network physics for enhanced agent learning. Further, a parallel training approach on multiple scenarios is employed to avoid biasing the agent to a few scenarios and make it robust to the natural variability in grid operations. Without these modifications to the training procedure, the RL agent failed for most test scenarios, illustrating the importance of properly integrating domain knowledge of physical systems for real-world RL learning. The agent was tested by RTE for the 2019 learning to run the power network challenge and was awarded the <inline-formula><tex-math notation="LaTeX">$2^{nd}$</tex-math></inline-formula> place in accuracy and <inline-formula><tex-math notation="LaTeX">$1^{st}$</tex-math></inline-formula> place in speed. The developed code is open-sourced for public use. Analysis of a simple system proves the enhancement in training RL-agents using the curriculum.

Original languageEnglish (US)
Pages (from-to)1-15
Number of pages15
JournalIEEE Transactions on Power Systems
StateAccepted/In press - 2022


  • actor-critic agents
  • Bars
  • cascading mitigation
  • L2RPN
  • open-sourced
  • parallel computing
  • Power grids
  • Power transmission lines
  • reinforcement learning
  • Substations
  • Switches
  • Topology
  • Training

ASJC Scopus subject areas

  • Energy Engineering and Power Technology
  • Electrical and Electronic Engineering


Dive into the research topics of 'Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading'. Together they form a unique fingerprint.

Cite this