Unified 2D and 3D Pre-Training of Molecular Representations
Proposes a unified 2D-3D molecular pretraining model, boosting property prediction accuracy by 8.3%.
Key Findings
Methodology
The approach integrates graph neural networks (GNNs) to encode both 2D molecular graphs and 3D conformations through three pretraining tasks: masked atom and coordinate reconstruction, 3D conformation generation conditioned on 2D graphs, and 2D graph generation conditioned on 3D conformations. The model employs MLM loss for masked reconstruction, permutation-invariant loss for symmetric substructures, and roto-translation invariant loss for 3D generation. It fuses atomic features, interatomic distances, and coordinates via multi-layer GN blocks, enabling comprehensive multi-modal representation learning.
Key Results
- On 11 downstream tasks, the model achieves SOTA on 10, with an average improvement of 8.3% over previous methods. Particularly, on 2D-only property prediction tasks, performance gains are substantial, especially in low-data regimes. For 3D conformation generation, the mean matching scores improve by 7.7% and 3.6%, respectively, surpassing prior approaches.
- Pretrained on the PCQM4Mv2 dataset (3.38 million molecules), the model demonstrates superior generalization across diverse molecular properties and conformations, validating the effectiveness of multi-modal fusion.
- Incorporating symmetry-aware permutation invariance and spatial invariance losses enhances robustness against structural variations, leading to better generalization in complex molecular scenarios.
Significance
This work pioneers a unified framework that simultaneously encodes and generates both 2D and 3D molecular information, addressing the limitations of prior single-modality models. It significantly advances molecular property prediction, conformer generation, and molecular understanding, with broad implications for drug discovery, materials science, and computational chemistry. The multi-task pretraining strategy offers a new paradigm for leveraging multimodal data, setting a foundation for future research in joint molecular modeling.
Technical Contribution
The paper introduces a novel multi-task learning architecture combining masked reconstruction, symmetry-aware conformer generation, and graph generation, all within a single GNN-based framework. It innovatively applies permutation and roto-translation invariant losses to handle structural symmetries and spatial transformations. The model's ability to process both 2D and 3D inputs end-to-end, with effective information fusion, distinguishes it from prior works that treat these modalities separately, providing a powerful tool for comprehensive molecular representation.
Novelty
This is the first approach to jointly encode and generate molecules' 2D graphs and 3D conformations within a single unified model. The integration of permutation-invariant and spatial-invariant losses for multi-modal pretraining is a key innovation, enabling robust learning of symmetric and spatially complex structures. Unlike previous methods that rely on separate encoders or mutual information maximization, this work achieves deep fusion through end-to-end training, marking a significant step forward in molecular deep learning.
Limitations
- The model's performance may degrade on highly symmetric or extremely large molecules due to limited data diversity and computational complexity. Its reliance on high-quality 3D conformations also poses data acquisition challenges. Additionally, inference speed for large datasets remains an issue, requiring further optimization. Handling dynamic or flexible molecules with multiple conformations is still an open challenge, necessitating future extensions.
- The current architecture primarily targets static conformations; extending to dynamic molecular behaviors or reaction pathways remains unexplored. Moreover, the computational cost of training and inference could hinder deployment in real-time applications. Addressing these limitations will be crucial for broader industrial adoption.
Future Work
Future directions include developing more efficient architectures to scale to larger molecular datasets, integrating dynamic conformational modeling, and reducing reliance on high-quality 3D data through semi-supervised or unsupervised learning. Exploring transfer learning across different molecular domains and incorporating experimental data for validation are also promising avenues. Ultimately, extending this framework to real-world drug discovery pipelines and materials design will be a key goal.
AI Executive Summary
Molecular representation learning has become a cornerstone in computational chemistry, enabling predictive modeling for drug discovery and materials science. Traditional approaches often focus on either 2D molecular graphs or 3D conformations, each capturing different aspects of molecular structure. However, these modalities are inherently complementary, and leveraging them jointly promises richer, more informative representations.
This paper introduces a pioneering unified framework that simultaneously encodes and generates both 2D and 3D molecular information within a single graph neural network (GNN) architecture. The core innovation lies in designing a multi-task pretraining strategy that includes masked atom and coordinate reconstruction, 3D conformation generation conditioned on 2D graphs, and 2D graph generation conditioned on 3D conformations. The model employs advanced loss functions—permutation-invariant for symmetric substructures and roto-translation invariant for spatial transformations—ensuring robustness and structural fidelity.
Pretrained on the large-scale PCQM4Mv2 dataset, the model demonstrates remarkable performance across a suite of downstream tasks. It achieves state-of-the-art results on 10 out of 11 molecular property prediction benchmarks, with an average improvement of 8.3%. In conformer generation tasks, the model surpasses previous methods with significant gains in matching scores, validating its capacity for accurate 3D structure synthesis.
The significance of this work extends beyond mere performance metrics. By effectively fusing multi-modal information, it addresses longstanding challenges in molecular modeling, such as symmetry handling and spatial invariance. Its potential applications span drug discovery, materials design, and reaction prediction, offering a versatile tool for scientists and industry practitioners.
Despite these advances, limitations remain, including computational costs and data dependency. Future research will focus on scaling, dynamic conformations, and integration into real-world pipelines. Overall, this work marks a substantial step forward in the quest for comprehensive molecular understanding through deep learning.
Deep Dive
Abstract
Molecular representation learning has attracted much attention recently. A molecule can be viewed as a 2D graph with nodes/atoms connected by edges/bonds, and can also be represented by a 3D conformation with 3-dimensional coordinates of all atoms. We note that most previous work handles 2D and 3D information separately, while jointly leveraging these two sources may foster a more informative representation. In this work, we explore this appealing idea and propose a new representation learning method based on a unified 2D and 3D pre-training. Atom coordinates and interatomic distances are encoded and then fused with atomic representations through graph neural networks. The model is pre-trained on three tasks: reconstruction of masked atoms and coordinates, 3D conformation generation conditioned on 2D graph, and 2D graph generation conditioned on 3D conformation. We evaluate our method on 11 downstream molecular property prediction tasks: 7 with 2D information only and 4 with both 2D and 3D information. Our method achieves state-of-the-art results on 10 tasks, and the average improvement on 2D-only tasks is 8.3%. Our method also achieves significant improvement on two 3D conformation generation tasks.