---
slug: release-msamp-v0.4
title: Releasing MS-AMP v0.4
author: Yuxiang Yang
author_title: MS-AMP Team
author_url: https://github.com/tocean
tags: [MS-AMP, announcement, release]
---

We are very happy to announce that **MS-AMP 0.4.0 version** is officially released today!

You can install and try MS-AMP by following [Getting Started Tutorial](https://azure.github.io/MS-AMP/docs/getting-started/installation).

## MS-AMP 0.4.0 Release Notes

### MS-AMP Improvements

- Improve GPT-3 performance by optimizing the FP8-gradient accumulation with kernel fusion technology
- Support FP8 in FSDP
- Support DeepSpeed+TE+MSAMP and add cifar10 example
- Support MSAMP+TE+DDP
- Update DeepSpeed to latest version
- Update TransformerEngin to V1.1 and flash-attn to latest version
- Support CUDA 12.2
- Fix several bugs in DeepSpeed integration

### MS-AMP-Examples Improvements

- Improve document for data processing in GPT3
- Add launch script for pretraining GPT-6b7
- Use new API of TransformerEngine in Megatron-LM

### Document Improvements

- Add docker usage in Installation page
- Tell customer how to run FSDP and DeepSpeed+TE+MSAMP  example in "Run Examples" page