Readings

Italicized resources are required, and other resources are suggested.

Table of contents

1. Introduction
   1. Deep Learning Review
2. Risk Analysis
3. Robustness
   1. Adversarial Robustness
   2. Long Tails and Distribution Shift
4. Monitoring
   1. OOD and Malicious Behavior Detection
   2. Interpretable Uncertainty
   3. Transparency
   4. Trojans
   5. Detecting and Forecasting Emergent Behavior
5. Control
   1. Power-Seeking
   2. Honest AI
   3. Machine Ethics
6. Systemic Safety
   1. Forecasting
   2. ML for Cyberdefense
   3. Cooperative AI
7. X-Risk

Introduction

• Unsolved Problems in ML Safety

• Concrete Problems in AI Safety

• Workshop On Safety And Control For Artificial Intelligence

Deep Learning Review

• Deep Residual Learning for Image Recognition

• Attention Is All You Need

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Decoupled Weight Decay Regularization

• Layer Normalization

• Gaussian Error Linear Units (GELUs)

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

• Adam: A Method for Stochastic Optimization

Risk Analysis

• Complex Systems and AI Safety

• Emergence (Intermediate)

• Introduction to STAMP (video, slides)

• Safe Design, Consequence assessment, Safety models

• Systemantics Appendix

• How Complex Systems Fail

Robustness

Adversarial Robustness

• Universal and Transferable Adversarial Attacks on Aligned Language Models

• Towards Deep Learning Models Resistant to Adversarial Attacks

• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks (website)

• Certified Adversarial Robustness via Randomized Smoothing

• Adversarial Examples Are a Natural Consequence of Test Error in Noise

• Certified Defenses against Adversarial Examples

• Towards Evaluating the Robustness of Neural Networks

Long Tails and Distribution Shift

• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

• PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures

• WILDS: A Benchmark of in-the-Wild Distribution Shifts

• ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

• Adversarial NLI: A New Benchmark for Natural Language Understanding

• Natural Adversarial Examples

• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Monitoring

OOD and Malicious Behavior Detection

• Deep Anomaly Detection with Outlier Exposure

• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks

• ViM: Out-Of-Distribution with Virtual-logit Matching

• VOS: Learning What You Don’t Know by Virtual Outlier Synthesis

• Scaling Out-of-Distribution Detection for Real-World Settings

• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Interpretable Uncertainty

• On Calibration of Modern Neural Networks

• Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

• PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures

• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

• Posterior calibration and exploratory analysis for natural language processing models

• Accurate Uncertainties for Deep Learning Using Calibrated Regression

Transparency

• Representation Engineering: A Top-Down Approach to AI Transparency

• In-context Learning and Induction Heads

• The Mythos of Model Interpretability

• Sanity Checks for Saliency Maps

• Locating and Editing Factual Knowledge in GPT

• Acquisition of Chess Knowledge in AlphaZero

• Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

• Convergent Learning: Do different neural networks learn the same representations?

Trojans

• Poisoning and Backdooring Contrastive Learning

• Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs

• Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks

• TrojAI

• Detecting AI Trojans Using Meta Neural Analysis

• STRIP: A Defence Against Trojan Attacks on Deep Neural Networks

• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning

• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Detecting and Forecasting Emergent Behavior

• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

• The Basic AI Drives

• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

• Optimal Policies Tend to Seek Power

• The Off-Switch Game

• Goal Misgeneralization in Deep Reinforcement Learning

Control

Power-Seeking

• Existential Risk from Power-Seeking AI

• Do the Rewards Justify the Means?

Honest AI

• Discovering Latent Knowledge in Language Models Without Supervision

• TruthfulQA: Measuring How Models Mimic Human Falsehoods

• Truthful AI: Developing and governing AI that does not lie

Machine Ethics

• What Would Jiminy Cricket Do? Towards Agents That Behave Morally

• Normative Ethics Introduction

• Ethics Background (Introduction through “Absolute Rights or Prima Facie Duties”)

• Aligning AI With Shared Human Values

• Avoiding Side Effects in Complex Environments

• Conservative Agency via Attainable Utility Preservation

• The Structure of Normative Ethics

Systemic Safety

Forecasting

• Forecasting Future World Events with Neural Networks

• On Single Point Forecasts for Fat-Tailed Variables

• On the Difference between Binary Prediction and True Exposure With Implications For Forecasting Tournaments and Decision Making Research

• Superforecasting – Philip Tetlock

ML for Cyberdefense

• Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions

Cooperative AI

• Uehiro Lectures 2022

• Open Problems in Cooperative AI

• Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda

X-Risk

• An Overview of Catastrophic AI Risks

• Can we build AI without losing control over it?

• What happens when our computers get smarter than we are?

• X-Risk Motivations for Safety Research Directions

• Natural Selection Favors AIs over Humans