The fast and accurate modelling of thermal errors in machining is an important aspect for the implementation of thermal error compensation. This paper presents a novel modelling approach for thermal error compensation on CNC machine tools. The method combines the Adaptive Neuro Fuzzy Inference System (ANFIS) and Grey system theory to predict thermal errors in machining. Instead of following a traditional approach, which utilises original data patterns to construct the ANFIS model, this paper proposes to exploit Accumulation Generation Operation (AGO) to simplify the modelling procedures. AGO, a basis of the Grey system theory, is used to uncover a development tendency so that the features and laws of integration hidden in the chaotic raw data can be sufficiently revealed. AGO properties make it easier for the proposed model to design and predict. According to the simulation results, the proposed model demonstrates stronger prediction power than standard ANFIS model only with minimum number of training samples.