Sequencing Initial Conditions in Physics-Informed Neural Networks
DOI:
https://doi.org/10.56946/jce.v3i1.345Keywords:
Scientific machine learning PINN, soft-regularization, multiphysics modeling, chemical engineering PDEsAbstract
The scientific machine learning (SciML) field has introduced a new class of models called physics-informed neural networks (PINNs). These models incorporate domain-specific knowledge as soft constraints on a loss function and use machine learning techniques to train the model. Although PINN models have shown promising results for simple problems, they are prone to failure when moderate level of complexities are added to the problems. We demonstrate that the existing baseline models, in particular PINN and evolutionary sampling (Evo), are unable to capture the solution to differential equations with convection, reaction, and diffusion operators when the imposed initial condition is non-trivial. We then propose a promising solution to address these types of failure modes. This approach involves coupling Curriculum learning with the baseline models, where the network first trains on PDEs with simple initial conditions and is progressively exposed to more complex initial conditions. Our results show that we can reduce the error by 1 – 2 orders of magnitude with our proposed method compared to regular PINN and Evo.
References
Acharya N. et al. Instability analysis of poiseuille flow between two parallel walls partially obstructed by porous surfaces. In APS Division of Fluid Dynamics Meeting Abstracts, 2019, pp. NP05-140.
Ainsworth M., Dong J., 2021. Galerkin neural networks: a framework for approximating variational equations with error control. SIAM J. Sci. Comput., 43, A2474-A2501. https://doi.org/10.1137/20M1366587
Aygün A. Physics informed neural networks for computational fluid dynamics. Middle East Technical University, Master's thesis, 2023.
Batra R. et al. Emerging Materials Intelligence Ecosystems Propelled by Machine Learning. Nature Reviews. Materials, 2021, 6(8), 655-678. https://doi.org/10.1038/s41578-020-00255-y
Beck C., Hutzenthaler M., Jentzen A., Kuckuck B.J., 2020. An overview on deep learning-based approximation methods for partial differential equations.
Bengio Y. et al. Curriculum learning. Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 41-48. https://doi.org/10.1145/1553374.1553380
Bharadwaja B.V.S.S. et al. Physics-Informed Machine Learning and Uncertainty Quantification for Mechanics of Heterogeneous Materials. Integrating Materials and Manufacturing Innovation, 2022, 11(4), 607-627. https://doi.org/10.1007/s40192-022-00283-2
Cai S. et al. Physics-informed neural networks (PINNs) for fluid mechanics: a review. Acta Mechanica Sinica/Lixue Xuebao, 2021, 37(12), p. 1727-1738. https://doi.org/10.1007/s10409-021-01148-1
Cai S., Wang Z., Wang S., Perdikaris P., Karniadakis G. E., 2021. "Physics-Informed Neural Networks for Heat Transfer Problems." ASME. J. Heat Transfer, June 2021; 143(6): 060801. https://doi.org/10.1115/1.4050542 https://doi.org/10.1115/1.4050542
Chen Y. et al. Physics-informed neural networks for inverse problems in nano-optics and metamaterials. Optics Express, 2020, 28(8), 11618-11633. https://doi.org/10.1364/OE.384875
Chervonyi Y. et al. Semi-analytical industrial cooling system model for reinforcement learning. arXiv preprint arXiv:2207.13131, 2022.
Cooley M. et al. Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Bases. OpenReview, 2023, https://openreview.net/forum?id=40Mw2GJnlZ.
Daw A. et al. Mitigating propagation failures in PINNs using evolutionary sampling. arXiv preprint, 2022, arXiv:.
Dekhovich A. et al. iPINNs: Incremental learning for Physics-informed neural networks. arXiv preprint arXiv:2304.04854, 2023.
Droniou J., Le N., 2020. The gradient discretization method for slow and fast diffusion porous media equations. SIAM Journal on Numerical Analysis, 58(3), pp: 1965-1992. https://doi.org/10.1137/19M1260165
Du H. et al. Modeling density-driven flow in porous media by physics-informed neural networks for CO2 sequestration. Computers and Geotechnics, 2023, 159, 105433. https://doi.org/10.1016/j.compgeo.2023.105433
Du H., Zhao Z., Cheng H. Modeling Density-Driven Flow in Porous Media by Physics-Informed Neural Networks for CO2 Sequestration. Computers and Geotechnics, 2023, 159, 105433. https://doi.org/10.1016/j.compgeo.2023.105433
Edwards C. Neural networks learn to speed up simulations. Communications of the ACM, 2022, 65(5), 27-29. https://doi.org/10.1145/3524015
Elahi A. et al. Temperature-Transferable Coarse-Grained Model for Poly(propylene oxide) to Study Thermo-Responsive Behavior of Triblock Copolymers. J. Phys. Chem. B, 2022, 126, p. 292-307. https://doi.org/10.1021/acs.jpcb.1c06318
Elahi A., Chaudhuri S. Computational Fluid Dynamics Modeling of the Filtration of 2D Materials Using Hollow Fiber Membranes. ChemEngineering, 2023, 7, 108. https://doi.org/10.3390/chemengineering7060108
Geneva N., Zabaras N. Modeling the dynamics of PDE systems with physics-constrained deep auto-regressive networks. Journal of Computational Physics, 2020, 403, 109056. https://doi.org/10.1016/j.jcp.2019.109056
Guo J. et al. Pre-training strategy for solving evolution equations based on physics-informed neural networks. arXiv preprint, 2022, arXiv:2212.00798. https://doi.org/10.1016/j.jcp.2023.112258
Gusmão G. S., Medford A. J., 2023. Maximum-likelihood Estimators in Physics-Informed Neural Networks for High-dimensional Inverse Problems. arXiv preprint arXiv:2304.05991. https://doi.org/10.1016/j.compchemeng.2023.108547
Hooshyar S. et al. Energy budget analysis of plane poiseuille-couette flow over a permeable surface. Bulletin of the American Physical Society, 2020.
Hooshyar S. et al. The impact of imposed Couette flow on the stability of pressure-driven flows over porous surfaces. Journal of Engineering Mathematics, 2022, 132(1). https://doi.org/10.1007/s10665-021-10195-3
https://doi.org/10.1016/j.ijheatmasstransfer.2020.119678
Hussaini M., Zang T.A., 1987. Spectral methods in fluid dynamics. Annual Review of Fluid Mechanics, 19(1), pp: 339-367. https://doi.org/10.1146/annurev.fluid.19.1.339
Jagtap A.D. et al. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 2020, 365, 113028. https://doi.org/10.1016/j.cma.2020.113028
Jin X. et al. NSFNNets (Navier-Stokes Flow Nets): Physics-informed neural networks for the incompressible Navier-Stokes equations. Journal of Computational Physics, 2021, 426, 109951. https://doi.org/10.1016/j.jcp.2020.109951
Karniadakis G.E., Kevrekidis I.G., Lu L., Perdikaris P., Wang S., Yang L., 2021. Physics-informed machine learning. Nature Reviews Physics, pp: 422-440. https://doi.org/10.1038/s42254-021-00314-5
Krishnapriyan A. et al. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 2021, 34, 26548-26560.
Mao Z. et al. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 2020, 360, 112789. https://doi.org/10.1016/j.cma.2019.112789
Mishra S., Molinaro R. Physics informed neural networks for simulating radiative transfer. Journal of Quantitative Spectroscopy and Radiative Transfer, 2021, 270, 107705. https://doi.org/10.1016/j.jqsrt.2021.107705
Münzer M., Bard C. A Curriculum-Training-Based Strategy for Distributing Collocation Points during Physics-Informed Neural Network Training. arXiv preprint, 2022, arXiv:2211.11396.
Ngo S.I., Lim Y. Forward Physics-Informed Neural Networks Suitable for Multiple Operating Conditions of Catalytic CO2 Methanation Isothermal Fixed-Bed. IFAC-PapersOnLine, 2022, 55(7), 429-434. https://doi.org/10.1016/j.ifacol.2022.07.481
Ngo S.I., Lim Y. Modeling density-driven flow in porous media by physics-informed neural networks for CO2 sequestration. Computer Aided Chemical Engineering, 2022, pp. 1675-1680. https://doi.org/10.1016/B978-0-323-85159-6.50279-7
Ngo S.I., Lim Y. Solution and Parameter Identification of a Fixed-Bed Reactor Model for Catalytic CO2 Methanation using Physics-Informed Neural Networks. Catalysts, 2021, 11(11), 1304. https://doi.org/10.3390/catal11111304
Pang G. et al. fPINNs: Fractional Physics-Informed Neural Networks. SIAM Journal on Scientific Computing, 2019, 41(4), A2603-A2626. https://doi.org/10.1137/18M1229845
Raissi M., Perdikaris P., Karniadakis G.E., 2019. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, pp: 686-707. https://doi.org/10.1016/j.jcp.2018.10.045
Sak H. et al. Modeling dependence dynamics of air pollution: Pollution risk simulation and prediction of PM2.5 levels. arXiv preprint, 2016, arXiv:1602.05349.
Shi S., Liu D., Huo Z. Simulation of flow field in silicon single-crystal growth using physics-informed neural network with spatial information. Physics of Fluids, 2022, 34(11), 113610. https://doi.org/10.1063/5.0123811
Shiratori S. et al. Spatio-temporal thickness variation and transient Marangoni number in striations during spin coating. International Journal of Heat and Mass Transfer, 2020, 154, 119678.
Shiratori S., Kubokawa T. Double-peaked edge-bead in drying film of solvent-resin mixtures. Physics of Fluids, 2015, 27, 102105. https://doi.org/10.1063/1.4934670
Stegmeir A., Ross A., Body T., Francisquez M., Zholobenko W., Coster D., Kang S., 2019. Global turbulence simulations of the tokamak edge region with GRILLIX. Physics of Plasmas, 26(5), 052517. https://doi.org/10.1063/1.5089864
Subramanian S. et al. Adaptive Self-supervision Algorithms for Physics-informed Neural Networks. arXiv preprint, 2022, arXiv:2207.04084.
Subramanian S. et al. Adaptive Self-supervision Algorithms for Physics-informed Neural Networks. arXiv preprint, 2022, arXiv:2207.04084.
Wang S. et al. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint, 2022, arXiv:2203.07404.
Wang S., Teng Y., Perdikaris P. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint, 2020, arXiv:2001.04536.
Yin M. et al. Non-invasive inference of thrombus material properties with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 2021, 375, 113603. https://doi.org/10.1016/j.cma.2020.113603
Zhu Q., Liu Z., Yan J. Machine Learning for Metal Additive Manufacturing: Predicting Temperature and Melt Pool Fluid Dynamics using Physics-Informed Neural Networks. Computational Mechanics, 2021, 67(2), 619-635. https://doi.org/10.1007/s00466-020-01952-9
Zienkiewicz O., Taylor R., Zhu J., 2005. The finite element method: its basis and fundamentals. Elsevier.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
