A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids

Decision-making of microgridsin the condition of a dynamic uncertain biddingenvironment has always been a significant subject of interest in the context of energy markets. The emerging application of reinforcement learning algorithms in energy marketsprovides solutions to this problem. In this paper, we investigate the potential of applying a Q-learningalgorithm into a continuous double auctionmechanism. By choosing a global supply and demand relationship as states and considering both bidding priceand quantity as actions, a new Q-learningarchitecture is proposed to better reflect personalized biddingpreferences and response to real-time marketconditions. The application of battery energy storage system performs an alternative form of demand responseby exerting potential capacity. A Q-cube framework is designed to describe the Q-valuedistribution iteration. Results from a case study on 14 microgridsin Guizhou Province, China indicate that the proposed Q-cube framework is capable of making rational biddingdecisions and raising the microgrids’ profits.