We present 2–4 GHz observations of polarized radio galaxies toward eight fast radio bursts (FRBs), producing grids of Faraday rotation measure (RM) sources with sky densities of 9–28 polarized sources per square degree. Using a Bayesian interpolation framework, we constrain Galactic RM fluctuations below ∼1 deg2 angular scales around the FRB positions. Despite the positions of all eight FRBs being located far from the Galactic plane, we constrain previously unresolved small-scale Galactic RM structures around six of the eight FRBs. In two of these fields, we find potential changes in the sign of the Galactic RM that are not captured by previous, sparsely sampled RM grid observations. Our Galactic RM estimate toward the FRBs differs by a few radians per square meter up to ∼40 rad m−2 from the all-sky Galactic RM map of S. Hutschenreuter et al. Extrapolating our results to the known population of polarized FRB sources, we may be incorrectly interpreting the host galaxy RM for ∼30% of the FRB source population with current RM grid observations. Measuring small-scale Galactic RM variations is crucial for identifying FRBs in low-density and weakly magnetized environments, which in turn could serve as potent probes of cosmic magnetism. This framework of reconstructing continuous Galactic RM structure from RM grid observations can be readily applied to FRBs that fall within the sky coverage of upcoming large-sky radio polarization surveys of radio galaxies, such as the Very Large Array Sky Survey and the Polarization Sky Survey of the Universe's Magnetism.