BEGIN:VCALENDAR
VERSION:2.0
PRODID:icalendar-ruby
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VTIMEZONE
TZID:Europe/Vienna
BEGIN:DAYLIGHT
DTSTART:20230326T030000
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
TZNAME:CEST
END:DAYLIGHT
BEGIN:STANDARD
DTSTART:20221030T020000
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
TZNAME:CET
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260424T052359Z
UID:64009151779e5993252290@ist.ac.at
DTSTART:20230321T130000
DTEND:20230321T150000
DESCRIPTION:Speaker: Max Ryabinin\nhosted by Dan Alistarh\nAbstract: Over t
 he recent years\, the scale of deep learning has increased dramatically: p
 retraining models like GPT-3 can cost millions of dollars\, and even their
  inference requires significant resources. In this talk\, I will present a
 n alternative approach: instead of using expensive clusters\, we can lever
 age the resources of volunteers or several organizations. I will cover sev
 eral papers addressing the challenges of such a setup published at NeurIPS
 '20\, '21\, and ICML'22\, as well as Hivemind  our open-source library for
  decentralized DL.I will also highlight SWARM Parallelism and Petals  our 
 latest works about decentralized pretraining and inference of large langua
 ge models. SWARM is a system for training large models over slow networks 
 of heterogeneous unreliable devices: to achieve this goal\, it relies on r
 andomized pipelines and dynamic rebalancing between pipeline stages. In tu
 rn\, Petals leverages the same techniques and allows everyone to run\, fin
 etune or inspect the internals of LLMs like BLOOM or OPT-175B without acce
 ss to many GPUs or the need for offloading  even from Colab notebooks.
LOCATION:Heinzel Seminar Room / Office Bldg West (I21.EG.101)\, ISTA
ORGANIZER:cfrancois@ist.ac.at
SUMMARY:Max Ryabinin: Decentralized Deep Learning: Training and Running Lar
 ge Models over the Internet
URL:https://talks-calendar.ista.ac.at/events/4069
END:VEVENT
END:VCALENDAR