Learning in data-scarce tabular domains: a comparative evaluation of tabular foundation models

In real-world machine learning applications, tabular data is the most prevalent dataformat. In almost all fields tabular data is used but still until recent years tabular learning has relied on gradient-boosted decision trees even when at the same time deep learning and pretrained foundation models have revolutionized image and text processing. Tabular data have many challenging characteristics compared to image or text data and there- fore the development of tabular foundation models has been slow, but in the resent years new promising methods based on in-context learning have been developed. This thesis focuses on three of them: TabPFNv2.5, TabICLv2 and ConTextTab. The experimental part of this thesis is conducted as comparative evaluation of performance and generalizability of tabular foun- dation models on classification and regression tasks. XGBoost and CatBoost were used as traditional methods in the experiments. The experiments focused mainly on scenarios where training data size is 5000 rows or smaller so that suitability of tabular foundation models to data-scarce domains could be assessed. The results obtained show that tabular foundation models are suitable for data-scarce domains, because in most situations tabular foundation models produced good predictions even when training set size was only 1000 - 3000 rows. Tabular foundation models also outperformed traditional models in almost all tests. Based on the results obtained the most promising tabular foundation model was TabICL.