This study presents a computational micromechanics approach based on the finite element method (FEM) to model heterogeneous, inelastic asphalt mixtures with rate-dependent fracture failure. The model accounts for the mixture heterogeneity as FEM meshes are generated from the digital images of actual specimens. The inelastic nature of the mixtures is modeled by including the viscoelastic constitutive relation of the material and the fracture process zone, which is modeled by the cohesive zone concept. A computational modeling framework and related experimental protocols are presented, and the applicability of the model is demonstrated through virtual testing simulations of asphalt concrete mixtures. Model simulations are discussed by comparing predictions to the laboratory test results. It is expected that the model with further improvements can provide better insights into the effects of mixture constituents on the overall mixture's performance, while reducing modeling efforts and achieving significant savings in experimental costs and time.