Embedding Numerical Features and Meta-Features in Tabular Deep Learning

Abstract

Tabular data is ubiquitous in real-world applications, and an increasing number of deep learning approaches have been developed for tabular data prediction. Among these approaches, embedding techniques serve as both a common and essential component. However, the design of tabular embedding paradigms remains relatively limited, and there is a lack of systematic evaluation regarding the performance of many existing methods in specific scenarios. In this paper, we focus on embedding numerical features and meta-features. To enrich the embedding methods for numerical features, we propose an ordering-oriented regularization technique applicable to piecewise linear embeddings, along with an unsupervised feature grouping method to facilitate partial embedding sharing. We demonstrate that these methods contribute to building more efficient and lightweight embedding modules. Importantly, we highlight ordering and sharing as two promising directions in the design of embeddings for numerical features. Additionally, we address several evaluation gaps: we assess the robustness of existing embeddings for numerical features and evaluate a set of general designs separately for data type embeddings and positional embeddings, providing insights into their practical applications and further developments.