Discovery of Innovative Polymers for Next-Generation Gas-Separation Membranes using Interpretable Machine Learning

Polymer membranes perform innumerable separations with far-reaching environmental implications. Despite decades of research on membrane technologies, design of new membrane materials remains a largely Edisonian process. To address this shortcoming, we demonstrate a generalizable, accurate machine-learning (ML) implementation for the discovery of innovative polymers with ideal separation performance. Specifically, multitask ML models are trained on available experimental data to link polymer chemistry to gas permeabilities of He, H 2 , O 2 , N 2 , CO 2 , and CH 4 . We interpret the ML models and extract chemical heuristics for membrane design, through Shapley Additive exPlanations (SHAP) analysis. We then screen over nine million hypothetical polymers through our models and identify thousands of candidates that lie well above current performance upper bounds. Notably, we discover hundreds of never-before-seen ultrapermeable polymer membranes with O 2 and CO 2 permeability greater than 10 4 and 10 5 Barrer, respectively. These hypothetical polymers are capable of overcoming undesirable trade-off relationship between permeability and selectivity, thus significantly expanding the currently limited library of polymer membranes for highly efficient gas separations. High-fidelity molecular dynamics simulations confirm the ML-predicted gas permeabilities of the promising candidates, which suggests that many can be translated to reality.


INTRODUCTION
Polymer membranes are a flexible, processable, and inexpensive platform to provide a myriad of separations that fill critical roles in climate change mitigation (carbon capture) and resiliency (water treatment). For gas separations, polymer membranes have been widely used in the separation of mixtures in many industrial processes, including oxygen enrichment 1 , biogas purification 2 , and post-combustion carbon capture 3 . In particular, carbon capture processes are garnering increased attention to reduce emissions to the environment, and membrane technologies offer known advantages such as high energy efficiency and operational simplicity due to flexibility and scalability [4][5][6] . In post combustion, pre-combustion, and oxy-combustion, CO2/N2, CO2/H2, and O2/N2 separations are respectively important for environmental conservation.
During membrane-based gas separation, a gas mixture is typically driven through a membrane by pressure where separation is achieved through differences in individual gas permeabilities 7 . The performance of membrane processes is determined by the membrane's permeability for a specific gas species, Pi, where i specifies the gas type. The membrane permeability is calculated from Fick's law of diffusion, | | Δ / , where Ji is the flux of gas i, and Δ is the pressure drop across a membrane of thickness l. When comparing the permeability of gas A with that of gas B, another performance measure is the membrane's selectivity between two gases, α, which is defined as / . An ideal membrane for a given binary gas separation would have high permeability and high selectivity. Increasing gas permeability and selectivity in these membranes would allow for more efficient industrial processes by increasing the process throughput, reducing energy costs, and achieving a purer product [8][9][10] . However, there is a well-known permeability-selectivity tradeoff for polymer gas-separation membranes 7 , which is defined by the Robeson upper bound 11  CO2/CH4 and CO2/N2 separations 12,13 ) that reflect improved membrane performance. Identifying new materials that break this upper bound has driven and continues to drive materials discovery efforts for membranes [14][15][16][17] .
Remarkably, in the decades of technological development in the membrane science field, design of new membrane materials has been, and remains, a largely trial-and-error process, guided by experience and intuition 18 . Current approaches generally involve tuning chemical groups to increase affinity and solubility towards a desired gas, or incorporating greater free volume to increase overall diffusivity 19 . When assembling a new polymer, typically a desired enhancement is targeted (i.e., higher CO2 affinity, higher overall permeability, aging resistance etc.) and a chemical group that is likely to achieve that enhancement is incorporated into the polymer chemistry [20][21][22][23] . For achieving higher permeability, polymers of intrinsic microporosity (PIMs) have been extensively studied during the last two decades 24,25 . PIMs generally enhance fractional free volume via inefficient chain packing to increase permeability, while simultaneously stiffening the polymer backbone and improving solubility selectivity [25][26][27] . Efforts to design improved chemistries for PIMs generally involve tuning the contortion group, increasing steric frustration via modifications to side chains, or further stiffening the polymer backbone [28][29][30][31] . Still, many of these studies remain limited to an Edisonian approach, unable to identify or utilize big-picture rules of chemistry-property relationships in polymer membranes.
Further complicating matters, synthesis of new polymeric materials and subsequent testing of permeance and selectivity is a time-consuming, expensive, and incomplete process that can miss high-performance candidates. Molecular modeling approaches, such as Monte Carlo/molecular dynamics (MC/MD) simulations, can reasonably predict a polymer membrane's gas permeabilities without costly experiments [32][33][34][35] . However, even these high throughput molecular simulations are too computationally expensive to explore the vast chemical space of polymers on the order of 10 6~1 0 10 . By contrast, simplified approximations to predict gas permeability for a given membrane are low cost but inaccurate. Most simply, group contribution methods sum together the gas permeability contribution of each chemical moiety in a polymer, but they do not necessarily consider connectivity and cannot expand into new classes of polymers 36 . Permeability can also be calculated via diffusivity based on the polymer's free volume and the solution-diffusion model of gas transport using various theoretical models, but these theories are incomplete [37][38][39] . In short, there is no efficient and accurate predictive model for gas permeability based on polymer-membrane chemistry.
Machine learning (ML) is a promising data-centric approach for prediction of gas permeabilities by learning a functional model based on polymer chemistry 40,41 . ML methods using chemical inputs have been successfully applied to accurately predicting many polymer properties including glass transition temperature [42][43][44] , thermal conductivity 45 , dielectric constants 46 , organic photovoltaic properties 47,48 , and transport properties [49][50][51]  To tackle the above challenges, we demonstrate interpretable, supervised ML models that can accurately predict the He, H2, O2, N2, CO2, and CH4 permeabilities of gas separation membranes based on polymer chemistry-as part of our ML-assisted discovery workflow outlined in Fig. 1.
Our training data consists of polymer chemistry and experimental gas permeabilities from two large databases, PolyInfo 52 and Membrane Society of Australasia (MSA) 53 , for hundreds of homopolymers, including PIMs. We utilize two representations for the polymer repeating unit, namely chemical descriptors as generated by RDKit 54 (listed in  57 analysis. Our analysis provides a chemical explanation for the well-known permeability-selectivity tradeoff in membranes, and many of the other physical insights that we draw are consistent with established membrane design principles.
Using the trained ML models, we perform high-throughput screening of over nine million hypothetical polymers with unknown permeabilities, including many polyimides and ladder polymers that can be classified as PIMs. Thus, we identify thousands of promising polymers for gas separation membranes with desirable performance, which lie well above 2008 Robeson upper bounds. Finally, we perform high-fidelity MD simulations to confirm that the ML-predicted permeabilities of top-performing polymers are very accurate. Overall, our ML-assisted workflow is a promising method for the discovery of innovative polymers for next-generation gas-separation membranes to advance energy and environmental sustainability.

Datasets and chemical space under exploration
Our training dataset, Dataset A, consists of 778 homopolymers (353 unique polymer chemistries), but not all entries have gas permeability data reported on all six gases under study: He, H2, O2, N2, CO2, and CH4. Dataset A is manually collected from the PoLyInfo database (experimental data from before 2005) and is merged with data from the MSA database (beyond 2005). As shown in Fig. 2(a-b), in general, the more recent MSA database contains polymers with higher permeability, e.g., CO2 permeability greater than 10 3 Barrers, and entries that surpass the 2008 Robeson upper bound. Fig. 2(c-d) also show that there is not a significant difference when missing gas permeabilities are imputed via extremely randomized trees (ERT) vs Bayesian linear regression (BLR). In these plots, we identify several known PIMs, with their corresponding chemical structures shown in Fig. 2 Hypothetical polymers generated based on existing ladder polymers Table 1 Summary of the datasets explored in this work. Dataset A is the training set, which contains polymers with known chemistries and permeabilities. Datasets B, C, and D contain hypothetical polymers with unknown permeabilities (used for screening and polymer discovery). They span three different chemical spaces: known polymers from PoLyInfo, polyimides, and ladder polymers, respectively.

Performance of ML models for gas permeability prediction
To quantify performance, we evaluate the accuracy and generalizability of our ML models, namely RFs and DNN ensembles trained on chemical descriptors and MFFs. For our supervised ML models, the metric of study is the R 2 correlation between the predicted and actual permeabilities on the training and test sets, as summarized in  Table 2 Summary of the performances of supervised ML models as scored by the R 2 value between the predicted and actual permeabilities. All ML models make multitask predictions for the six gas permeabilities of He, H 2 , O 2 , N 2 , CO 2 , and CH 4 and are trained on the data that is augmented using BLR imputation. The DNN ensemble models perform better than the RF models, and models trained on MFFs perform slightly better than models trained on molecular descriptors.

Physical insights from interpretation of ML models
Usually, ML models are treated as black boxes, which makes it challenging to understand any physical principles learned by the models. However, we find that obtaining SHAP values from our ML models on chemical descriptors and MFFs makes our models not only accurate but also interpretable. By extracting the most important chemical features that predict gas permeability, we draw physical insights into the molecular design of polymer membranes. Here we decide to focus our analysis on the DNN ensemble because of its better performance, but other model types can also be explained using the same method.   In Fig. 4, we perform the same type of feature importance analysis using SHAP values, for the DNN model trained on MFFs. Here, we highlight the most important chemical substructures in the prediction of gas permeability. As shown in Fig. 4(a), the most important substructure overall is 2854, the methyl group. We believe that this feature facilitates permeability because it is hydrophobic and its shape contributes to steric frustration between polymer chains. Similarly, the quaternary carbon connected to an aliphatic ring (substructure 2168) contributes to increasing permeability, which supports our findings above. The DNN model also learns that the number of connection bonds, substructure 1781, is correlated with gas permeability, because many highpermeability PIMs are ladder polymers with four connection points per repeating unit, as opposed to two for a typical polymer. The correlation matrix between chemical substructures (Fig. S9 of Supporting Information) suggests that most of the important substructure features are independent of one another. However, substructures 1781, 1432, and 822 are highly correlated and all have a positive relation to gas permeability (Fig. 4(b)). Upon closer examination, we find that substructure 1781 is contained within substructure 1432, which is contained in substructure 822.
Substructures 1432 and 822, two double-bonded carbons connected to an aromatic ring, define polyacetylenes, which demonstrate some of the highest permeabilities among non-porous polymers in gas separations 66 . By contrast, polar groups generally have negative contributions to gas permeability, as shown in Fig. 4(b). For example, double-bonded oxygens (799 and 2706), ethers (1519), and nitrogen atoms (2906) are all inversely correlated with gas permeability. Since most gas molecules are non-polar, the presence of these polar groups generally reduces the solubility of gases, which explains the negative effect.
However, in Fig. 4(a), our ML models show that these groups tend to have a greater negative impact (measured by SHAP value) on N2 and CH4 permeability, compared to O2 and CO2, which explains why the presence of these groups in polyimides, ladder polymers, and poly(ethylene oxides) 19 can increase selectivity by widening the permeability difference between certain pairs of gases, which is desirable for gas separations. This supports a known heuristic in membrane design, that CO2 selectivity can be increased via increased CO2 solubility by incorporating oxygen atoms into polymer membranes 19,23 . Across the board for substructures, SHAP values tend to be higher for N2 and CH4 compared to O2 and CO2 (Fig. 4(a)), which suggests that incorporating chemistries that increase permeability (i.e. methyl groups) is likely to come at the cost of selectivity. Our ML models thus elucidate a chemical basis for the permeability/selectivity tradeoff: chemical features that increase permeability are likely to do so to a greater extent for molecules that are less permeable (N2 and CH4), but chemical features that reduce permeability are also likely to impact these molecules to a greater magnitude-thereby increasing selectivity for the more permeable gas (O2 and CO2). Achieving high permeability and selectivity thus becomes a balancing act. This unique understanding is unlocked from the ability of ML to learn complex patterns in data.

Discovery of high-performance polymers and validation through MD simulations
After training our RF and DNN ensemble ML models, we use the models based on MFFs for high-throughput screening and discovery of high-performance polymers for gas separations. We choose the ML models using MFFs for simplicity due to their slightly better performance and lower memory requirements. We calculate MFFs for millions of hypothetical polymers in Datasets Broadly, we find that the RF model predicts permeabilities in a much narrower space than the DNN ensemble, which explains its lower R 2 values on the test set and further supports the observation that the DNN ensemble is more accurate and generalizable. Predicted permeabilities for each screening dataset lie in their expected region in the permeability-selectivity space, which further supports the accuracy of our ML models. Namely, both models predict permeabilities close to existing Robeson upper bounds for polymers in Dataset D, which consists entirely of ladder polymers (a subclass of PIMs). Similarly, Dataset C consists of polyimides (including many PIMs), and their permeability predictions span a space that includes polymers below and above the Robeson upper bound, reflecting the dataset's diversity. However, Dataset B corresponds to mostly polymers with low permeability and selectivity. We believe that this can be explained by the fact that PoLyInfo is a broad database that contains many polymers that are not suitable for gas separation applications, and Dataset B is populated from existing polymers in PoLyInfo. Most promisingly, the DNN model predicts thousands of polymers from Dataset C to be above the 2008 Robeson upper bound, for O2/N2, CO2/CH4, CO2/N2, and H2/CO2 separations, which is summarized in Table 3. We further find that the DNN ensemble trained on MFFs not only generalizes but also extrapolates. We discover a class of hypothetical polymers in Dataset C with never-before-seen ultrahigh CO2 permeability (greater than 10 5 Barrer) and a class of polymers with ultrahigh O2 permeability (greater than 10 4 Barrer)-even though our training set only contains 12 polymers with O2 permeability greater than 10 4 Barrer and only 2 polymers with CO2 permeability greater than 10 5 Barrer.  Both the RF and DNN model can identify polymers with high performance in Datasets C and D. The chemical structures of the selected polymers are drawn in Fig. 6(a). We highlight some of the top substructures identified from SHAP analysis (Fig. 4) in these chemical structures, which corroborates our earlier conclusions. For instance, the higher permeability polymers tend to have more methyl groups (substructure 2854) and methyl groups attached to aliphatic rings (substructure 2168) to increase steric frustration. Meanwhile, double-bonded oxygens (substructure 2706) in the polyimide backbone help to maintain selectivity for gases such as O2 and CO2.
As shown in Fig. 6(b-e), ML-predicted performances lie very close to their respective MDsimulated performances for separations involving O2, N2, CO2, CH4, and H2. Error ranges for permeability calculations from simulations and predictions from ML models are provided in  can each be formed through the polycondensation of a known diamine and a dianhydride, as given in Fig. S13-14 of Supporting Information.
To further investigate the superior permeability of the selected top polymer candidates, we generate their realistic structural models and analyze their pore structures via molecular simulations in Fig. S17 of Supporting Information. Details of our simulated polymerization algorithm are given in the Methods section (and additionally Fig. S12-16 of Supporting Information). In comparison with PIM-1, our top candidates have more voids, enhanced microporosity, and larger pore radii. The pore size distribution of the top candidate polymers is wider and shifted to the right, further suggesting enhanced microporosity and permeability.

DISCUSSION
In this work, we demonstrate an accurate and cost-effective ML implementation that can effectively explore the ever-expanding design space for polymeric gas-separation membrane materials, by learning their synthesis-property relationships. Firstly, our study reveals that fixed chemical descriptors or fingerprints are both excellent representations for predicting gas permeabilities of polymer membranes. Corroborating our recent benchmark study on polymer glass-transition temperature 44 , we conclude that the choice of chemical representation generally plays a limited role in each ML model's performance, as long as sufficient chemical substructures are captured. Additional features, such as microstructure, could be considered in future ML models, given the importance of microstructural characteristics such as FVEs in solution-diffusion transport theory of membranes 67 . Incorporation of such characteristics as input features has improved metal-organic framework adsorption prediction 68,69 , compared to using solely chemical descriptors. These microstructural features could be efficiently calculated via MD simulations, as being demonstrated in this work. Alternatively, because high-throughput MD simulations can calculate gas permeabilities with reasonable accuracy, these simulations could also be used to augment the training set or be incorporated into active learning frameworks to reduce the uncertainty of ML models 71 . Nevertheless, we find that using fixed chemical features captures sufficient information to predict the gas permeabilities of the polymer membranes studied here.
We additionally gain insight into how the choice of ML model affects performance. At the same time, we demonstrate that ensembling is a powerful technique for improving prediction accuracy while simultaneously quantifying uncertainty. Traditionally, RF models are thought to work better on small datasets, while deep learning is reserved for large training sets. But while decision trees are adequate for capturing simple relationships, neural networks can in principle approximate any function to arbitrary accuracy 72 . In our study, we demonstrate that deep learning can be effectively applied to small training datasets on the order of a few hundred training samples.
We believe that our DNN method is accurate for two reasons. Firstly, a DNN that is deep enough will not overfit if it's in the "modern" interpolating regime 73 . Secondly, each DNN model, seeing limited data, captures complexities and nuances in the data, which results in individual predictions with high variance; however, the overall model generalizes well when predictions are averaged together via ensembling 74 . Importantly, training the 16 DNNs in our study and evaluating predictions for millions of samples is still computationally tractable from a cost standpoint.
Though various other neural networks such as graph neural networks are garnering increased interest for certain molecular discovery and synthesis tasks 75 , we do not observe notable performance gains from training graph convolutional, recurrent, or convolutional neural networks.
We have reached a similar conclusion from our polymer informatics benchmark study on polymer glass transition 44 . In short, we believe that deep learning techniques, even standard multilayer perceptrons, have much broader applicability to small datasets of chemical features than previously assumed.
We further show that SHAP analysis can succinctly elucidate the impacts of input features, which erodes the paradigm that ML models are black boxes 76 . SHAP values can be calculated for nearly all supervised ML models, and we encourage future chemical and polymer informatics studies to take advantage of explainability in ML 77 . A recent study also used coloring of substructures when training a graph neural network for interpretable ML 78 , which suggests that feature-importance analysis of ML models can be extended beyond fixed representations to learned chemical representations.
Our study of fixed feature importance solidifies many existing membrane design principles, but additionally offers unique, generalized guidance for the molecular engineering of new polymers for gas separations. Overall, SHAP analysis illuminates the chemical balancing act required for overcoming the permeability/selectivity tradeoff. Polymers must juggle (A) the number of bulky chemical moieties-i.e. methyl groups, aliphatic rings-that increase microporosity (permeability at the expense of selectivity) with (B) the number of polar groups-i.e. carbonyls, oxygens-that increase relative CO2 and O2 affinity (selectivity at the expense of permeability). P-DNN-C3 and C4 are case studies into this balancing act. They achieve high permeability primarily through methyl groups and large aliphatic rings, which is a relatively underexplored strategy in membrane design. At the same time, these polymers attain unprecedented O2/N2 selectivity via the incorporation of polar groups, such as carbonyls and sulfonyls. By contrast, P-DNN-C1 and C2 each feature an inflexible polycyclic backbone and two trifluoromethyl containing side chains. Amazingly, they demonstrate that our DNN model learns the importance of bulky spherical groups (such as trifluoromethyl groups) for creating steric frustration 79 , which has been recognized in the gas separation community as favoring higher gas permeability 25,80,81 .
Restricted backbone mobility plus the presence of the bulky pendant groups disrupts polymer chain packing and leads to high fractional free volume and ultrahigh permeability, all while the polar polyimide backbone helps to maintain selectivity.
Differently, the discovered polymers from Dataset D utilize a rigid ladder-type backbone with a spirobifluorene (SBF) unit, like many other ladder-type PIMs 31 . The fused benzene rings in the SBF unit reduce the flexibility of the backbone around the spirocenter, and the two-bond ladder connections restrict the rotation ability of the backbone. The reduced chain flexibility may also prohibit chain motion to help resist physical aging 25 . To further increase permeability, P-DNN-D1 and P-DNN-D2 attach a fused tetramethyltetrahydronaphthalene (TMN) to the SBF unit, which incorporates additional aliphatic rings and methyl substituents 30 .
Overall, the generalizable ML models presented here are capable of efficiently discovering promising polymers with high performance, with thousands of candidates lying beyond the 2008 Robeson upper bound 11 . Additionally, the ultra-high permeability polymers discovered in this work would allow for never-before-seen industrial gas separations with higher throughput while maintaining sufficient selectivity. Incredibly, the DNN model can extrapolate relatively accurately to high permeability predictions that it did not see in training. We believe that this amazing performance primarily arises from careful selection of diverse training samples and training with a neural network that can capture complexities but also generalizations through multitask parameter sharing and ensembling.
Our experimentally validated MD simulations of gas permeability confirm the ML predictions, which suggests that many of the polymer candidates discovered here can be translated to reality in experiments. As elaborated upon in Fig. S2 of Supporting Information, each of the promising polyimides identified here have a well-defined cross-linking formation from existing PubChem chemicals, which makes their syntheses tractable. However, the difficulty of synthesizing complex polymers in a solution-processable manner should not be underestimated. Therefore, to facilitate the overcoming of this challenge, we have tabulated the thousands of promising polymers that we have identified and included them in the GitHub repository associated with this work (https://github.com/jsunn-y/PolymerGasMembraneML), which we encourage experimental and computational researchers to explore further. While our models consider membrane performance to be constant, future efforts should also take into account how aging, plasticization, and swelling can degrade membrane performance over time, which is an important consideration in membrane design 8,82 .
Ultimately, we provide the membrane design community with many novel high-performance polymer candidates and key chemical features to consider when designing their molecular structures. Many of the concepts demonstrated here can likely be extended to other materials discovery and design tasks, such as polymer membranes for desalination and water treatment 14 , high-temperature fuel cells 83 , and catalysis 84 . With the continual improvement of ML techniques and an increase in computing power, we expect that ML discovery frameworks will only gain popularity and deliver increasingly substantial results in materials discovery for a wide range of applications 85 .

Calculation of chemical representations for polymers
The workflow for our ML method to learn synthesis-property relationships of gas separation membranes is shown in Fig. 1 databases. In the datasets, each polymer entry is identified based on its unique SMILES string, a notation for chemical structures that represents a molecule as a unique string of ASCII characters 86 .
ML models for prediction of gas permeability from chemistry must utilize a descriptive and appropriate input to represent the polymer 44  The uncertainty of the prediction can then be measured as the variance between model outputs.

∈ℰ
While there are many ways to perform ensembling, we choose to use bootstrapping, or training each model in the ensemble with a different random subset of the training data. First, we randomly select 20% of the data to be the holdout set, which is used for performance scoring. 16 independent models are trained, using 80% of the entries in the non-holdout set each time, selected at random.
The training of our DNN ensemble on MFFs with BLR imputation of permeabilities is given in Alongside the models trained in Step 3 of our workflow, we can perform explainable ML. To strengthen our physical understanding of how chemical features are linked to performance in gas separation membranes, our primary tool involves assessing SHAP values from each model 57 . In essence, the SHAP approach considers how well a model performs when each feature is neglected during training. By analyzing the quantitative impact of leaving out a feature on the model prediction, a feature importance can be assigned. Moreover, each sample's impact on the final model prediction can also be evaluated.
Once the ML models are trained and achieve good performance, we then screen over nine million hypothetical polymers (summarized in Table 1) to predict their gas permeabilities, in Step 4. Our screening predictions are then used to identify promising polymer candidates with high permeability and selectivity. The code and datasets for our ML implementation can be found at https://github.com/jsunn-y/PolymerGasMembraneML.

Permeability validation using MD simulations
In Step 5, to validate the gas permeabilities of selected polymeric membranes, all-atom MD simulations, using Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) 89 , are performed to calculate each gas's permeability as the product of its solubility and diffusivity 34 : The polymer consistent force field (PCFF) [90][91][92] is employed to describe the interatomic interactions of both polymer and gas, which has been widely used to calculate the mechanical properties, cohesive energies, heat capacities and elastic constants of organic polymers 91,93,94 .
We construct the polymeric membrane models via the multi-step crosslinking of binary where N is the number of gas molecules and is the position of molecule at time . MSD is calculated from the ensemble average ⟨ ⋯ ⟩ of the trajectory, and we use the multiple-origin method to improve the statistical accuracy. In addition, to account for molecular adsorption to the polymer membrane at the saturation state, we consider the diffusivity of different numbers of gas molecules: 5, 10, 20, 30, 40, 50, and 100. We find that using 20, 30, or 40 gas molecules result in similar diffusivities, which are averaged to give the calculated diffusivity. Finally, based on the solution-diffusion mechanism, gas permeability ( ) in a polymer membrane can be expressed as the product of the diffusivity ( ) and the solubility constant ( ). As summarized in Table S7 of Supporting Information, our benchmark study on the solubility, diffusivity, and permeability of five pertinent gases (H2, N2, O2, CO2, and CH4) in a PIM-1 membrane agrees well with available experimental data and simulation results 33,61 , which suggests that our MD model and method are physically reasonable.