Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play
Por um escritor misterioso
Last updated 09 julho 2024
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://d3i71xaburhd42.cloudfront.net/b125c8933d0264b9a103cb8fa80f226f8c9c3cdc/5-Figure4-1.png)
Results indicate that, at least for relatively simple games such as 6x6 Othello and Connect Four, optimizing the sum, as AlphaZero does, performs consistently worse than other objectives, in particular by optimizing only the value loss. Recently, AlphaZero has achieved outstanding performance in playing Go, Chess, and Shogi. Players in AlphaZero consist of a combination of Monte Carlo Tree Search and a Deep Q-network, that is trained using self-play. The unified Deep Q-network has a policy-head and a value-head. In AlphaZero, during training, the optimization minimizes the sum of the policy loss and the value loss. However, it is not clear if and under which circumstances other formulations of the objective function are better. Therefore, in this paper, we perform experiments with combinations of these two optimization targets. Self-play is a computationally intensive method. By using small games, we are able to perform multiple test cases. We use a light-weight open source reimplementation of AlphaZero on two different games. We investigate optimizing the two targets independently, and also try different combinations (sum and product). Our results indicate that, at least for relatively simple games such as 6x6 Othello and Connect Four, optimizing the sum, as AlphaZero does, performs consistently worse than other objectives, in particular by optimizing only the value loss. Moreover, we find that care must be taken in computing the playing strength. Tournament Elo ratings differ from training Elo ratings—training Elo ratings, though cheap to compute and frequently reported, can be misleading and may lead to bias. It is currently not clear how these results transfer to more complex games and if there is a phase transition between our setting and the AlphaZero application to Go where the sum is seemingly the better choice.
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://miro.medium.com/v2/resize:fit:1202/1*ROq9V2D5eR_dDFFFfjA5zw.png)
How to build your own AlphaZero AI using Python and Keras
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://miro.medium.com/v2/resize:fit:1400/1*bb2ZcYKT-obLE4dinnQTmg.png)
Superhuman Algorithm: MuZero Explained, by Ape Gainz
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://www.mdpi.com/electronics/electronics-10-01533/article_deploy/html/images/electronics-10-01533-g001-550.jpg)
Electronics, Free Full-Text
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fnature24270/MediaObjects/41586_2017_Article_BFnature24270_Fig1_HTML.jpg)
Mastering the game of Go without human knowledge
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://d3i71xaburhd42.cloudfront.net/b125c8933d0264b9a103cb8fa80f226f8c9c3cdc/6-Figure6-1.png)
Policy or Value ? Loss Function and Playing Strength in AlphaZero
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://d3i71xaburhd42.cloudfront.net/b125c8933d0264b9a103cb8fa80f226f8c9c3cdc/5-Figure4-1.png)
Policy or Value ? Loss Function and Playing Strength in AlphaZero
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://media.arxiv-vanity.com/render-output/7241976/crownjewel.png)
Multiplayer AlphaZero – arXiv Vanity
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://media.arxiv-vanity.com/render-output/7909095/x1.png)
Representation Matters: The Game of Chess Poses a Challenge to
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://image.slidesharecdn.com/alphazeropresentationjournalclub-190811050615/85/alphazero-a-general-reinforcement-learning-algorithm-that-masters-chess-shogi-and-go-through-selfplay-3-320.jpg?cb=1668399516)
AlphaZero: A General Reinforcement Learning Algorithm that Masters
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://www.science.org/cms/10.1126/science.aar6404/asset/7e65d303-4d48-4ec2-9299-bbe101eecb88/assets/graphic/362_1140_f1.jpeg)
A general reinforcement learning algorithm that masters chess
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs00521-021-05928-5/MediaObjects/521_2021_5928_Fig8_HTML.png)
Value targets in off-policy AlphaZero: a new greedy backup
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://en.chessbase.com/Portals/all/2017/_eng/misc/alphazero-chess02.jpg)
The future is here – AlphaZero learns chess
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://miro.medium.com/v2/resize:fit:1400/1*T5AhtjSUat_6VbkN5bsquw.png)
AlphaZero, a novel Reinforcement Learning Algorithm, in JavaScript
![Policy or Value ? Loss Function and Playing Strength in AlphaZero-like Self-play](https://media.arxiv-vanity.com/render-output/7909095/x3.png)
Representation Matters: The Game of Chess Poses a Challenge to
Recomendado para você
-
AlphaZero, Vladimir Kramnik and reinventing chess09 julho 2024
-
AlphaZero Explained09 julho 2024
-
AlphaZero - Notes on AI09 julho 2024
-
AlphaGo Zero Explained In One Diagram, by David Foster, Applied Data Science09 julho 2024
-
Electronics, Free Full-Text09 julho 2024
-
AlphaZero: A General Reinforcement Learning Algorithm that Masters Chess, Shogi and Go through Self-Play09 julho 2024
-
AlphaZero-Inspired Game Learning: Faster Training by Using MCTS Only at Test Time09 julho 2024
-
AI Summary: Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search09 julho 2024
-
Training AlphaZero for 700,000 steps. Elo ratings were computed from09 julho 2024
-
AlphaZero paper peer-reviewed is available · Issue #2069 · leela-zero/leela-zero · GitHub09 julho 2024
você pode gostar
-
One Piece Film Gold Pocket Watch Compass09 julho 2024
-
Google permitirá anúncios de jogos NFT a partir de 15 de setembro09 julho 2024
-
Epic Space Adventures (LEGO Star Wars: Activity Book with Minifigure)09 julho 2024
-
Dragon Ball Xenoverse 2 ganha expansão com novos conteúdos09 julho 2024
-
NOVOS CÓDIGOS +TODOS OS CÓDIGOS DE RESET STATUS DO BLOX FRUITS09 julho 2024
-
La selección femenina de Uruguay se prepara para enfrentar a Puerto Rico en la fecha FIFA09 julho 2024
-
Shrek Meme Trippy Dance Hallucination GIF09 julho 2024
-
Mobile APK Download 2023 - Free - 9Apps09 julho 2024
-
Stick fight by zayyad1453 on DeviantArt09 julho 2024
-
Google ganha atualização com melhores momentos e game da Copa do09 julho 2024