Transformers for NLP 第一章

第一章 What are Transformers?


这一章,作者认为Transformers 是工业4.0这个观点进行了阐述。

An Industry 4.0 project manager can go to OpenAI’s cloud platform, sign up, obtain an API key, and get to work in a few minutes. A user can then enter a text, specify the NLP task, and obtain a response sent by a GPT-3 transformer engine. Finally, a user can go to GPT-3 Codex and create applications with no knowledge of programming. Prompt engineering is a new skill that emerged from these models.

首先是云平台,然后提供API访问 包括像Codex都是,只要大模型提供了云API 都算是工业4.0 。 工业4.0 其实是数据驱动的人工智能型的网络化“智能工厂”,确实现在类似的API,其实还没有进入工业领域。

围绕现在transformers这个模型,其实建立起来的是foudation models,算是模型基础设施,后面这种基础设施也只有大厂会来做。

AI 的新范式,感觉是以基础大模型+微调 组成的。 这确实算是一种新的颠覆。




工业4.0 AI的角色

  1. API – 大模型
  2. 库 – 开源库
  3. 训练和微调 – 提供计算,训练和微调服务
  4. 开发技巧 – 模型落地

Transformers for NLP笔记

版本:Transformers for Natural Language Processing_Build, train, and fine-tune deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, and GPT-3 2nd Edition



  • Chapter 1: What are Transformers?
  • Chapter 2: Getting Started with the Architecture of the Transformer Model
  • Chapter 3: Fine-Tuning BERT Models
  • Chapter 4: Pretraining a RoBERTa Model from Scratch
  • Chapter 5: Downstream NLP Tasks with Transformers
  • Chapter 6: Machine Translation with the Transformer
  • Chapter 7: The Rise of Suprahuman Transformers with GPT-3 Engines
  • Chapter 8: Applying Transformers to Legal and Financial Documents for AI Text Summarization
  • Chapter 9: Matching Tokenizers and Datasets
  • Chapter 10: Semantic Role Labeling with BERT-Based Transformers
  • Chapter 11: Let Your Data Do the Talking: Story, Questions, and Answers
  • Chapter 12: Detecting Customer Emotions to Make Predictions
  • Chapter 13: Analyzing Fake News with Transformers
  • Chapter 14: Interpreting Black Box Transformer Models
  • Chapter 15: From NLP to Task-Agnostic Transformer Models
  • Chapter 16: The Emergence of Transformer-Driven Copilots