The double ballbar (DBB) test is a well-known way to check the geometric error of axis interaction. The DBB test captures actual data from multiple error origins. Here, we define the DBB measurement result as the sinusoid error map model plus noise. Using this concept, we extract a single source geometric error value from the DBB error map by LS fitting. We considered the "noise" as mix error from other sources. To ensure the quality of a numerical fitting, we used a sinusoid model of each geometric error that was generated by simulation of axis movement based on homogeneous transformation matrices (HTMs) as general best-fit curve. To verify the proposed method, we extract a well-known geometric error of linear axes and compare it with the result from a commercial measurement system. This method is applicable to both a full circle and a truncated DBB test path. Then, we use the method to estimate the geometric error of axis interaction between linear and rotary axes in a five-axis machine. A sequence of DBB tests is arranged based on linear-linear and linear-rotary simultaneous motions. The tests contain seven DBB test runs with two setups, and are able to identify eleven geometry errors of interaction of axes in less time, and with less human "intervention" error.