Class Time: Mondays and Wednesdays 11:40am-12:55am

Location: 324 Milbank (Barnard College)

**Instructor: Smaranda Muresan**

Office: Milstein Center, 901 (Barnard College)

Office hours: Monday 1:30-2:30pm

Email: [email protected]

TA: Arkadiy Saakyan ****

Office: Milstein Center, 911A

Office hours: Thursdays 4-5pm

Email: [email protected]

📜 Course Description

Large Language Models (LLMs) such as GPT-3, ChatGPT, LLaMA are models that are trained on large amounts of data and are adaptable to a wide range of tasks. They are the basis of most state-of-the-art systems in Natural Language Processing. While the potential of these technologies for social good is large, the risks are also comparable. In this course, the students will learn the fundamentals about the modeling, theory and ethical aspects of LLMs and their applications, while gaining experience working with them. The course will be structured as a seminar, where one class is dedicated to instructor-led lecture and one to student-led discussion of papers around topics covered in the lecture. Each paper discussion will be structured as a panel of 3-4 students, each with an assigned role. Each panel role covers one aspect of critically assessing an academic/industry paper. Everyone in the class should participate by commenting and asking questions from the panel. The class is project-based, meaning there will be a semester-long project focused on evaluating LLMs and/or building LLMs around a topic/problem/task you care about, with an end of semester final paper. The projects will be done by groups of 3-4 students.

Prerequisite(s): COMS W3134 or W3136 or W3137 (or equivalent).

Announcements

<aside> 💡

[Due 1/31/2024 11:59 pm EST] Please select the paper for discussion via the sign up sheet shared in class and courseworks. You will choose 2 papers and choose a different role for each.

</aside>

<aside> 💡 [Due 01/24/2025 11:59pm EST] Please fill in the First Class Survey for the class even if you are on the waitlist and also even if you filled in the form to sign up for class. We would like to understand in more detail your background, skills and what you hope to learn from this class.

</aside>

🗓 Course Schedule

Some topics/readings might be subject to change

Untitled

🏆 Coursework and Grading

This is a seminar-type class, with two main activities:

• Paper Discussion: For each class, we will assign one or two required readings to discuss in class. The discussion of the papers will be lead by students, and we will conduct it as a role-playing discussion, where each presenter will be assigned a role (see below). This is inspired by Alec Jacobson and Colin Raffel’s Role-Playing paper reading seminars (https://colinraffel.com/blog/role-playing-seminar.html). Each student will sign up for 2 paper and choose different roles for each paper (see below). You cannot choose a wild card role more than once.

• Semester-long Project: There is a final project that you will be working on throughout the semester. You are strongly encouraged to form teams of 3-4 ****students for the final project.

  Paper Discussion as Role-Playing (30%)

  • Author of the paper [10min]: You are among the original authors of the paper. You’ll give a 10min short, high-quality presentation at a conference presenting the main contributions of the paper. Some paper will be more technical than others, you should focus on high level aspects not details. Here is some good advice from Mike Ernst. Rule of thumb (1 slide per minute). (See Ziems et al 2022 outstanding paper award at ACL 2022, **Inducing Positive Perspectives with Text Reframing,** it includes link to video**).**
  • Scientific Reviewer [5min]: The paper has not been published yet and is currently under review at a top conference where you have been assigned as a peer reviewer. Complete a full review of this paper answering all prompts of the official review form of the top venue in this research area (e.g., ACL/NAACL/EMNLP). For example, such as review form for *ACL conference is here (**https://aclrollingreview.org/reviewform**). This includes recommending whether to accept or reject the paper. Please maintain a polite and considerate voice when sharing limitations about the work (as if the readers of the reviews might be yourself!) and even better if you can suggest constructive and concrete action items to improve the paper. Please read carefully guidelines on how to write a good review in this tutorial (see Task 3).
  • Archeologist [5min]: This paper was found buried under ground in the desert. You are an archeologist who must determine where this paper sits in the context of previous and subsequent work. Find and report on one older paper cited within the current paper that substantially influenced the current paper and one newer paper that cites this current paper. You can also assess the impact in terms of citations (e.g., Google Scholar). You can also assess the paper in light of the current study that investigates citation amnesia in NLP field (Singh et al., 2023)
  • Wild card role [5 min]: Besides the three main roles above, we will discuss one “wild card role”. A student can choose a wild card role only once. Depending on the type of paper you can choose a role that you’d like to discuss such as
    • Replicability enthusiast: You want to implement the approach described in the paper as faithfully as possible. Are the methods and experimental conditions described in sufficient detail to be replicated? Were usable resources (code, data, etc.) released? If so, spend 30 min. looking through them to see how well documented they are and how easy they will be to use. Are the models open or closed. Is the training data documented, is the test data available
    • Social impact inspector: Identify how this paper self-assesses its (likely positive) impact on the world. Have any additional positive social impacts left out? What are possible negative social impacts that were overlooked or omitted?
    • Researcher: You are a researcher who is working on a new project in this area. Propose an imaginary follow-up project not just based on the current but only possible due to the existence and success of the current paper
    • Science Communicator/Illustrator: You are a tech journalist that would like to disseminate the scientific content of the article to a wider audience. You can write a short description of the main technical advances (without hype 😊) and/or propose a visual illustration of the technical approach.

  Semester-Long Project (60%)

  Projects will be conducted in teams of 3-4 students.

  • Project Types

    • Reproducing and potentially extending research paper published at ACL/EMNLP/NAACL/NeurIPs
    • Evaluating a variety of LLMs w.r.t some issues (e.g., reasoning, bias, toxicity, factuality, multilingual aspects)
    • Using LLMs to solve new tasks/problems
    • Using human-LLM collaboration to build new datasets for new tasks/problems
    • ….

  • Project Timeline and Grading

    • Choose team members (0%): Feb 10
    • Short Project proposal (5%): February 26
      • 1 page single column that includes your team, the type of project you choose, topic overall main idea. Goal here is for the instruction team to understand if your project aligns with the learning goals of the class and is feasible.
    • Project midterm report (20%): April 2
      • A short report (e.g 2-4 pages double column format (e.g., ACL style files)) that shows intermediate progress on your project. For example, if you tackle a problem like evaluating bias in LLMs, do a literature review and identify datasets and models you’d like to evaluate. This will allow the instruction team to give you feedback early on the direction of your project and also adjust scope if needed. If you are doing a reproducibility project: describe the paper you are tackling, the datasets and models (mention whether models are available etc).
    • Final Presentation (10%): April 30/May 5.
      • in-class short presentation (10min) presenting your project.
    • Final Report (25%): May 9
      • a final report (8 page double column (e.g. ACL style files), that presents your project. Each project should have an ethics section, a limitation sections (more details will be provided later).

    Class Participation (10%)

    • Filling out surveys (intake and mid-course); participate in class discussion when you are not presenting

  Learning Outcomes

  • Understand the basic conceptual and technical details of the building blocks of frontier LLMs.
  • Have the ability to implement and/or experiment with LLMs.
  • Understand the promises and limits of the abilities of existing frontier LLMs
  • Have a clearer picture of the landscape of the modern day, cutting-edge NLP research areas, and the broader societal impact of LLMs.
  • Have a complete experience in discussing and conducting NLP research, learning basic research abilities such as reading and critically assessing research papers, iterating on research ideas, conducting a literature search, implementing and/or experimenting with models, presenting results, and writing reports.

  Center for Accessibility Resources & Disability Services

  If you believe you may encounter barriers to the academic environment due to a documented disability or emerging health challenges, please feel free to contact me and/or the Center for Accessibility Resources & Disability Services (CARDS). Any student with approved academic accommodations is encouraged to contact me during office hours or via email. If you have questions regarding registering a disability or receiving accommodations for the semester, please contact CARDS at (212)8544634, [email protected], or learn more at barnard.edu/disability services. CARDS is located in 101 Altschul Hall.