Agent-Based Models in Power Systems – A Literature Review

In the last two decades, power systems have experienced several changes, mainly related to organizational and operational restructuring. The appearance of new actors contributes to developing new business models and modifies its traditional operation activities. As a direct result, there is a need for new control solutions and strategies to integrate these different players. Agent-Based Models (ABM) have been increasingly used to model complex systems since they are especially suited to model systems influenced by social interactions between flexible, autonomous, and proactive agents. This paper provides a review of the literature regarding ABM in power systems followed by an analysis in more detail regarding specific applications that are becoming relevant in this new paradigm. Author


Introduction
The transition of power systems from vertically integrated structures to deregulated markets and the implementation of new rules and business models originated the need for new operation and modeling strategy techniques. There are also under development new processes, mechanisms and equipment, such as demand response (DR), smart grids, storage systems, electrical vehicles (EV), energy communities, among others, which have been termed by some authors as the "democratization of energy" (Reis, Lopes, and Antunes 2018). With these new concepts, citizens can become producers and consumers (prosumers), so that various technical, social, economic, and environmental challenges should now be addressed. The complexity of incorporating new actors into the operation of power systems implies greater coordination among all stakeholders and leads to a new operational paradigm that requires innovative or adaptive methods to provide adequate decision support. This process should start with the decomposition of each player into smaller components, represented by individual agents, to perform actions to meet individual goals but also considering the behavior of the other participants and their impact on the overall system (Macal and North 2010). Thus, considering the operation of power systems with the participation of these new players, rather than just looking at the overall picture, makes the problem-solving in this domain an increasingly complex task. Agent-Based-Models (ABMs) can be considered a suitable tool to address this complexity. ABM has been proposed by many researchers as a proper modeling approach for complex, socio-technical problems (Bonabeau 2002). Mainly, ABM techniques have been applied to model and study electricity systems and markets and gained an increasing recognition (Ventosa et al. 2005). According to the development of these applications, this paper presents a literature review on ABM in Power Systems. The basic principles and a literature review of ABM in Power Systems is detailed in this work. It includes a survey considering scientific research over the last few years. Different applications of ABM in power systems are presented as well as the main conclusions.

Basic Principle of Agent-Based Models
As systems are becoming more complex, new tools, simulation, and modeling approaches are needed. An alternative to typical simulation techniques (such as traditional optimization techniques, discrete-event simulation, and differential equations) are ABM. An ABM is a computational model integrating individual and autonomous agents and their collective behavior. An autonomous agent acts on its own without external direction in response to situations the agent encounters during the simulation (Heath, Hill, and Ciarallo 2009). The following definitions of ABM are provided by Macal and North (2010), Pyka and Grebel (2006), and Gilbert (2008): -"Agent-based modeling is a way to model the dynamics of complex systems and complex adaptive systems. Such systems often self-organize and create emergent order. ABM also includes models of behavior (human or otherwise) and is used to observe agent behaviors and interactions' collective effects. The development of agent modeling tools, the availability of microdata and advances in computation have made possible a growing number of agent-based applications across a variety of domains and disciplines" (Macal and North 2010). -"The ABM approach consists of a decentralized collection of agents acting autonomously in various contexts. The massively parallel and local interactions can give rise to path dependencies, dynamic returns, and interaction. In an environment global phenomena such as the development and diffusion of technologies, the emergence of networks, herd-behavior, among others, which cause the transformation of the observed system can be modeled adequately. This modeling approach focuses on depicting the agents, their relationships and the processes governing the transformation" (Pyka and Grebel 2006). -"Formally, agent-based modeling is a computational method that enables a researcher to create, analyze, and experiment with models composed of agents that interact within an environment" (Gilbert 2008). ABM focuses on the modeling and simulation of complex systems, at a local level through the definition of their elementary units and, at a higher level, suited to model adaptive heterogeneous actors -agents. Shalizi (2006) defined an agent as a persistent thing that has some state worth representing and interacts with other agents, mutually modifying each other's states. Another definition of an agent was provided by Wooldridge and Jennings (1995) and indicates that an agent is a computer system situated in some environment, and capable of autonomous action in this environment in order to meet its design objectives. So, an agent is an entity situated in some environment that can autonomously react to changes in that environment. Besides, the environment is everything external to the agent and must be observable to or alterable by the agent (Wooldridge and Jennings 1995). Each agent chooses its strategy based on its previous experiences with other agents and through interaction with the environment, which helps him improving its decisions by modifying their strategies. The three basic concepts of ABM are: 1) Agents: can be a computer code, which can perform some tasks autonomously in a particular environment. The main features of an agent could include autonomy, reactivity, social ability, and pro-activity. In power systems, generators, ancillary service providers, protective devices, system operators, consumers, regulators, and retailers could be the agents; 2) Artifacts: are the components of the systems that are passive and are developed, shared, modified, developed, modified and utilized by the agents to carry out their activities competitively or cooperatively. Examples of artifacts in power systems could be transmission and distribution lines; 3) Workspaces: accommodates the agents and artifacts. It helps to define the topology of the environment and the idea of locality.  (Hager, Rauh, and Rid 2015), geographical information systems (el Raoui, Oudani, and Alaoui 2018), pandemics (Jalayer, Orsenigo, and Vercellis 2020), among others. With the transition from vertically integrated utilities to deregulated electricity markets, power systems' complexity is increasing, namely because new rules and players are emerging and being implemented. Distributed generation (smart grid and microgrids) (Farhangi 2010), EVs (Tan, Ramachandaramurthy, and Yong 2016), consumption flexibility and DR processes (Faria, Spínola, and Vale 2016), large penetration of renewable-based generation (Calabria, Saraiva, and Rocha 2016), energy efficiency measures (Zhou et al. 2016), and building energy management (Mittal et al. 2019), among many others, contribute to the increasing management and operation complexity, the transmission and distribution networks and the interactions between traditional and new players. The uncertainties associated with the renewable-based generation, electricity market prices, energy consumption, or EVs are just a few examples of the increased sources of uncertainty and thus of complexity brought to the power and the energy sector. In the last years, research has been devoted to deal with this complexity and address the challenges brought by this new paradigm. In this scope, we have looked at scientific research reported in the Web of Science database (WoS, n.d.) using different search terms to analyze the publications produced on this area and provide an overview of the recent research. Table  1 shows the overview of research documents produced since 2010 that were analyzed by different search terms. The search terms have been combined with similar variants within the power systems field to observe the results as accurately as possible. We looked for each search term in the abstract, title, and keywords of the papers. Also, we investigate the most cited papers in each search criterion and the most noticeable research authors by the number of articles produced.  A total of 1304 documents between 2010 and 2020 were found in the Web of Science database by using the search term "ABM" AND "agent based model*" in the abstract, title, and keywords. The 3 most cited papers aren't related to power systems and electrical engineering. This suggests the coverage of this topic in different fields and the innovative character of applying such a paradigm in power systems. Figure 1 presents the distribution of ABM among different scientific areas. The search term "ABM" AND "agent-based model" AND Power System* produced only 59 results. This suggests that the marriage between agent-based models and power systems is far from being mature. This was expected given how ABM have been maturing conceptually during the past few years. The most cited paper (Sornette 2014), with 118 citations, is not directly associated with power systems but with power-law distributions. It presents the different perspectives embraced in theories developed in financial economics compared with physics. However, the second most cited (Kuznetsova et al. 2014), with 117 citations, presents a microgrid energy management framework to optimize the microgrid stakeholder's individual objectives. This framework is exemplified considering a microgrid connected to an external grid via a transformer and includes a middle-size train station with an integrated photovoltaic power production system, a small energy production plant composed of urban wind turbines, and a surrounding district with residences and small businesses. The system is described by an ABM, in which each player is modeled as an individual agent aiming at a particular goal, (i) decreasing its expenses for power purchase or (ii) increasing its revenues from power selling (Kuznetsova et al. 2014).

ABM in Power Systems -Literature Review
The author with most papers published in this search field is Saraiva, Joao Tome, with 4 articles (Sousa and Saraiva 2017;Calabria, Saraiva, and Rocha 2015a, 2015b. In Sousa and Saraiva (2017), an ABM model is described using Q-learning to provide knowledge for the agents to select their decision. This model is designed to mimic the main features of the common electricity market between Portugal and Spain, the MIBEL. Apart from describing the developed model, this paper also includes preliminary results from its application to the MIBEL case. Calabria, Saraiva, and Rocha (2016) propose and test a bid based short-term market in order to overcome difficulties identified in the Brazilian electricity market. To simulate the behavior of the hydro units, it was implemented an ABM using the reinforcement Q-Learning algorithm, Simulated Annealing, and linear programming. A market design based on virtual reservoirs was proposed in Calabria, Saraiva, and Rocha (2015b). This model aims at enhancing the flexibility to enable market participants to comply with their contracts while still ensuring the efficient use of the water and maintaining the current security of supply. Some of the problems related to the current Brazilian electricity market were analyzed in Calabria, Saraiva, and Rocha (2015a). This paper proposes a new market design to overcome these issues based on the concept of virtual reservoirs and aims at enhancing the flexibility to enable market participants to comply with their contracts while still ensuring the efficient use of the energy resources and maintaining the current security supply level. ABM simulates the behavior of the market participants in this new framework. A more refined and focused search, based on search terms "ABM" AND "agent-based model" AND Power System* AND (market* OR electrical), produced only 19 results. However, a refined search was done as the first two results weren't directly related with power systems, which is the major topic. So, the search was done excluding the words "physics" and "tourism" providing now 17 results. The most cited paper in these research areas (Kuznetsova et al. 2015), provides an extended analysis of a microgrid energy management framework based on Robust Optimization (RO). The system is described by an ABM where each stakeholder is modeled as an individual agent. Each of these agents aim at optimizing a specific goal, either in terms of decreasing its expenses from power purchasing or by increasing its revenues coming from selling power. A systematic review of the potential of ABM to understand energy transactions from a socialscientific perspective is described in Hansen, Liu, and Morrison (2019). Six topic areas were identified, addressing different components of the energy system: Electricity Market, Consumption Dynamics/Consumer Behavior, Policy and Planning, New Technologies/Innovation, Energy System, Transitions. Chaudhari et al. (2019) present an ABM approach that considers an optimal EV charging infrastructure, taking into account several factors, such as the driver behavior, the location of charging stations, the electricity pricing. rules and no adaptation exists. Thus, agents in EMCAS have a lower adaptation capability than those in SEPIA. Moreover, the adaptation in EMCAS is restricted to generation company agents. Despite several references to ABM applied to power systems, the available models do not adequately consider a number of relevant issues such as the large presence in some systems of hydro stations and its reversal pumping feature and the large share of zero marginal cost intermittent technologies. In Sousa et al. (2013) it is discussed and proposed a conceptual model following the agent paradigm that deals with the inherent complexity of electricity markets such as, the Portuguese/Spanish Electricity Market (MIBEL). One of the tools that can be used to perform the management of a hydrothermal electric power system at a national level or with cross-border interconnections is VALORAGUA model. This tool establishes the optimal operation strategy for a given power system by using the "value of water" concept, in each power station, for each time interval (i.e. month/week) and each hydrological scenario. The model supplies detailed information on the technical, economic, and environmental behavior of each generation centre and the system. This model also computes thermal-based power generation emissions and optimizes the maintenance schedule of power plants. VALORAGUA is often used to: analyze energy import/export contracts; maximize power generation revenues; manage on the long-term the stored water in reservoirs with regulating capability; define the adequate use of water in multi-purpose units, considering its operation constraints (da Silva 2013). Multi Agent-based Electricity Market (MASCEM) is a simulator developed at the Polytechnic of Porto, Portugal (Praça et al. 2003) to study competitive electricity markets. The agents in MASCEM include a market facilitator, generators, consumers, market operators, traders, and a network operator. Their strategies are adequate to gain the highest possible advantage from each market context, acting in forward, day-ahead, and balancing markets and considering both simple and complex bids.

Smart grids
With the increase in the number of EVs and DR customers, ABMs can be a potential framework to challenging model problems in smart grids. Agents decide to buy, sell or store electricity depending on their demand, generation, and storage capacity. Zhou, Zhao, and Wang (2011) studied the impact of the level of participation of the commercial building in DR programs. It also examined how price-based DR can improve the efficiency and reliability of electricity systems. A prototype ABM to examine the effects of individual behavior and social learning on electricity use patterns is presented in Snape, Irvine and Rynikiewicz (2011). This paper provides a holistic view on the electricity system considering technical aspects, human interaction, and framework policies. Chassin, Fuller, and Djilali (2014) present a flexible power system modeling tool using an agent-based approach to simulate smart grid paradigms, such as demand response, energy storage, retail markets, electric vehicles, and new automated distribution systems. Dave, Sooriyabandara, and Yearworth (2013) present a business idea where the DR potential of households through aggregators is exploited. These authors detail that peak load reductions can be obtained using this approach. The potential large-scale introduction of EVs is another relevant aspect of future smart grids. EVs provide an interesting potential to control electricity demand in an intelligent way given the significant load-shifting options. A stochastic model for mobility behavior and ABM simulation tool is presented in Dallinger and Wietschel (2012).
In Foti and Vavalis (2015), a learning approach for strategic consumers in smart electricity markets has been designed. A machine learning approach and its integration with a widely used energy simulation platform was proposed. Yasir et al. (2015) present an ABM architecture for coordinating locally-connected microgrids, thereby supporting more cost-effective integration into the main power grid. The interconnected microgrids, with renewable energy sources and energy storage devices, employ agents so that each microgrid can choose to save or resell its stored energy in an open market in order to optimize its utility and revenues.

Energy communities
An agent-based approach to model zero energy communities is described in Mittal et al. (2019). This paper details a conceptual ABM for an urban neighborhood to predict the household level of adopting renewable energy behaviors in the presence of multiple options. Reis et al. (2019) model a community of residential prosumer agents that individually optimize the energy use to minimize energy costs and dissatisfaction. Each residential prosumer is modeled as an individual agent with specific energy needs and preferences. Reis, Lopes, and Antunes (2018) present a modeling approach to simulate energy trading between two energy community members capable of exchanging information and energy between them. A community of residential prosumer agents is modelled in Reis et al. (2019). Each residential prosumer is represented by an individual agent with specific energy needs and preferences. The residential agents exchange information with a coordinator agent that provides community resources or power purchased from the grid if needed. At the coordination level several optimization processes are performed to optimize the community resources and at each prosumer level to minimize agents' costs and dissatisfaction.

Energy storage
Energy storage is a promising technologic development for many contemporary issues in electricity markets and power systems operation. It could be configured in terms of Virtual Power Plants (VPP) and enable energy interaction between multiple PV prosumers under the form of direct sharing and buffered sharing (Liu et al. 2018). Meanwhile, as an intermediary between PV prosumers and the utility grid, VPP coordinates each prosumer's energy consumption behavior by setting internal prices and conducts power transactions with the utility grid. A payment scheme to compensate EVs customers that participate in a VPP is presented in Vasirani et al. (2013). VPPs are considered as coalitions of wind generators and EVs, where wind generators seek to use EVs as a storage devices to deal with the variations of wind generation. The simulation model provided in Praça et al. (2003) introduced VPPs in ABM. Another topic of interest is storage in the form of hydro reservoirs. This aspect is detailed and discussed in Saraiva (2015, 2017) in the context of MIBEL. This model is used to define adequate generation/pumping schedules and to assess the impact of these operation strategies in the market prices, therefore, passing from a simple price taker approach to a more complex and realistic price maker model.

Conclusions
In recent years, power systems are moving to a more complex operation paradigm with an increasing number of players and more complex and participated processes. In this context, the use of ABM to model and simulate these new problems and agents is becoming increasingly attractive. However, such development is far from being mature due to the innovative character of applying such a paradigm to power systems problems. This work presents and describes ABM characteristics supported with a literature review addressing how and when ABM should be applied. A survey on the most recent papers associated with ABM is provided, namely an in-depth overview on ABM applied to power systems, where the most cited papers, as well as the most noticeable research authors, are enumerated and presented. Finally, some relevant literature focusing on power system operating paradigms with ABM modeling approach is detailed in this work, specifically smart grids, electricity markets, energy communities, and energy storage systems. In conclusion, the application of ABM to power systems is far from being mature, meaning that the exploitation of common strategies developed with these tools regarding this new paradigm is a topic to explore in a future research.