Tutorial: (RL4CAS) Reinforcement Learning for Computer Architecture and Systems Research

Abstract

Motivation: Reinforcement learning has been demonstrated as a powerful and promising tool in many scientific and gaming problems. Recently, RL has shown promising results to solve problems in the area of computer architecture and compiler. Nevertheless, the lack of an easily accessible reinforcement learning framework and tutorial has prevented its full potential in solving architectural, security, and optimization problems.

This Tutorial: We aim to make RL accessible to computer architecture and system researchers, lowering the barrier for researchers to use RL in their own domain specific problems. The tutorial will be a combination of lectures/talks and hands-on labs. We will use our recent research artifacts as an example to demonstrate the effectiveness of RL in computer architecture and system research.

First, we will give a general introduction on the foundations of RL, in which we explain the RL concept, the RL environment (gym), and a RL training framework. Then we will discuss the application of RL in computer systems. The first example is CompilerGym, which improves the performance of code using RL-curated compiler flags. The second example is AutoCAT, where the RL agent is used to explore the potential cache-timing attacks. We will provide the detailed RL formulation, the example code and a lab section to give the audience hands-on experience on how to solve a computer system problem using RL. Below is the tentative schedule for the tutorial.

Target Audience: The target audience include but not limited to computer architecture and systems (attending ISCA, and other conference attendees at FCRC) researchers who are interested in machine learning methods and their applications in architecture and systems, applied machine learning researchers who are exploring new applications of machine learning in domain-specific problems. Graduate and undergraduate students who are interested in computer systems and machine learning in general are especially suitable for this tutorial as the tutorial provides hands-on coding experience with RL.

Expected Outcome: This tutorial will create concrete examples and hands on experience to the audience on how to formulate a computer architecture and system problem as an RL problem, and run RL training. We hope the attendance will proactively use RL in their own research problems after the tutorial.

This tutorial

The inaugural RL4CAS tutorial will be held at ISCA 2023 in Orlando on Saturaday, June 17th, 2023, 9am-12:30pm. We will also enable livestream by Zoom. The speakers include Yuandong Tian, Chris Cummins, Vincent Moens, Mulong Luo and Wenjie Xiong.

Time	Topic	Speaker(s)
9:00-9:05a	Welcoming Remarks [Video][slides]
9:05-9:40a	Introduction to RL and its applications to systems [slides] [Video]	Yuandong Tian, Meta AI
9:40-10:15a	CompilerGym: RL for compiler optimization [slides] [Video]	Chris Cummins, Meta AI
10:15-10:45a	RL for computer architecture and security [slides] [Video]	Wenjie Xiong, Virginia Tech
10:45-11:00a	Introduction to TorchRL, an RL Library [Slides] [Video]	Vincent Moens, Meta AI
11:00-11:20a	Coffee Break
11:20-12:30a	Lab: hands on with RL for compArch security [docs][Video]	Mulong Luo, Cornell and Vincent Moens, Meta AI

Contact us

For any further questions please contact ml2558@cornell.edu

About us

	Organizer
	Mulong Luo (Cornell) currently a final-year Ph.D. candidate at Cornell University advised by Prof. Edward Suh. His research interests are in computer architecture, resilience and reliability, and machine learning.
	Wenjie Xiong (Virginia Tech) is an assistant professor at Virginia Tech. Her research interests are in computer architecture and hardware security. Her work was selected as an Honorable Mention of IEEE Micro Top Picks 2021 and the featured paper of IEEE Transactions on Computers (TC).
	Yuandong Tian (Meta AI) is a research scientist and senior manager in Meta AI Research, working on reinforcement learning, representation learning and optimization. He is the first-author recipient of 2021 ICML Outstanding Paper Honorable Mentions and 2013 ICCV Marr Prize Honorable Mentions, and is the lead scientist and engineer for the OpenGo project. He also received the 2022 CGO Distinguished Paper Award.
	Hsien-Hsin Sean Lee (Intel) is an Intel Fellow leading the heterogeneous system technologies team in the Software and Advanced Technology Group where he focuses on the inter-domain innovation and co-optimization across computer architecture, middleware, machine learning, and semiconductor technologies. He is a Fellow of the IEEE.
	G. Edward Suh (Cornell/Meta AI) is a professor of ECE at Cornell University and a Research Scientist at Meta AI. His research interests include computer systems in general with particular focus on computer architecture and security. His current research focuses on building secure computing systems for secure and private machine learning, and using machine learning to improve the security of computer systems. His past work on the AEGIS secure processor is recognized with the test of time award by Intel for its contribution for trusted execution environments deployed across the industry today. He is a Fellow of IEEE.