Introducing CM3Leon - a powerful, retrieval-augmented, token-based, decoder-only multi-modal language model. It excels in generating and infilling both text and images. CM3Leon leverages the CM3 multi-modal architecture and showcases the significant advantages of scaling up and fine-tuning using diverse instruction-style data. It stands out as the first multi-modal model trained with a recipe adapted from text-only language models. We present CM3Leon, a retrieval-augmented, tokenbased, decoder-only multi-modal language model capable of generating and infilling both text and images. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pretraining stage and a second multi-task supervised fine-tuning (SFT) stage. CM3Leon achieves state-of-the art performance in text-to-image generation with 5 times less training compute than comparable methods (zero-shot MS-COCO FID of 4.88).

Modern deep learning is now a scale-driven game, where larger models, empowered by increased compute and data, dominate benchmarks. Our goal is to provide possibly best performing model for a given comput. To this end we try to fundamentally improving scalability through three approaches: Retrieval, Boosting, and Diffusion Augmented Transformers.

We provide a BART-like model: BARTSmiles, pre-trained on molecular representations. We quantitatively show that when applied to the molecular domain, the BART objective learns representations that implicitly encode our downstream tasks of interest. BARTSmiles consistently outperforms other self-supervised representations across classification, regression, and generation tasks setting a new SOTA on 11 tasks. Lastly, we show that standard attribution interpretability methods, when applied to BARTSmiles, highlight certain substructures that chemists use to explain specific properties of molecules.

We provide systems for en-ru and en-de language pairs for the WMT20 Biomedical Machine Translation shared task. For the en- ru pair, our submissions achieve the best BLEU scores, with en→ru direction outperforming the other systems by a significant margin. We explain most of the improvements by our heavy data preprocessing pipeline which attempts to fix poorly aligned sentences in the parallel data.

The beam emittance measurement at low energies is an important issue to optimize the facility performance and compensate the space charge effects. In this paper, we present the results of electron beam emittance measurement for AREAL beam energy of 2.5 MeV using the quadrupole scan technique.