Feature engineering with Seldon

If you have spent any time building models on tabular data, feature engineering has a certain shape to it. You have a gradient-boosted tree (XGBoost, LightGBM, take your pick), the standard tool for the job, and most of your effort goes into feeding it. Interaction terms, target encodings, rolling aggregates, count features, the lags and the ratios. A lot of that work is done on the model's behalf: a tree can only split on what you give it, so you spend your days manufacturing things for it to split on.

The first regression tree, a method called AID, was published in 1963, and its descendants became the workhorses of tabular prediction: CART, random forests, the gradient-boosted machines we reach for today. Sixty-odd years is a lot of accumulated muscle memory, and most of what we know about preparing data for a model, we learned in its service. A tabular foundation model works on a different principle, and the old reflexes come along whether they still fit or not.

None of that is wasted, exactly. Features still matter, obviously, they are most of where the signal comes from, and a foundation model does not change that. What it changes is everything around them. There is no model to train and no hyperparameters to tune, so the slow machinery that used to sit between a feature idea and an answer is gone. Two things are left in front of you: the features, which you still engineer by hand, and a second lever you never had with a trained model. This piece is about both.

What you stop tuning

The model is Seldon, the tabular foundation model we have built at Neuralk. Whether it predicts more accurately than a well-tuned tree is a fair question, and one we have taken seriously in a technical report. Here I want to focus on something else it changes: what it removes from the loop. You don't train it. You give it a batch of examples with the answers already filled in, plus the rows you want predicted, and it returns predictions in a single call. Nothing is fitted, nothing is tuned.

With a tree you have two things to work with, the features and the forty-odd settings on the model itself, and people spend a startling share of their lives on the second set. Take those away and the surface you are optimising collapses to two levers you actually control: which features you show the model, and which rows you show it as examples. It sounds like less to work with, but it concentrates the mind on the part that was always doing the real work.

The bigger effect is on speed, in the loop where you actually test features. Training a gradient-boosted model is cheap on a toy table and slow on a real one: a million rows with a hundred features takes around twelve minutes to fit on a CPU, and production tables are often larger than that, and that is only one fit. Tuning means fitting it again and again, thirty to a hundred times for a normal hyperparameter search, and an honest walk-forward or cross-validation repeats the whole thing for every fold. Fits times trials times folds is where the hours go, and every new feature you want to try pays that bill again.

Seldon has no fit step and no hyperparameters to search. A stock-market backtest on US regional banks, 220 weekly predictions across four years, finishes in about 75 seconds, on the order of forty full feature experiments an hour.

Lever one: shape the features like the answer

The example that made the point for us was a signal on KRE, the SPDR regional-banking ETF, about 148 US regional banks. For each bank, every week, we wanted to predict the part of next month's return that is specific to the bank rather than the part that comes from the whole sector moving. Quants call that the residual return: take the bank's return, subtract the slice explained by the index moving (scaled by how sensitive the bank usually is to it), and what is left is the bank's own story. Suppose Truist gains 4% while the sector is flat. Almost all of that 4% is Truist, because the sector didn't carry it anywhere. That residual is what we were trying to predict.

We started with the standard kit: momentum, short-term reversal, volatility, how sensitive the bank is to the sector, size. Ordinary price features, the ones any quant would reach for. They worked, sort of.

Then came the obvious question: if we are predicting the bank-specific residual return, why feed the model whole-sector price movements? So we built features shaped like the target instead: the same residual return, measured backwards over the past 1, 3, 6 and 12 months. The same shape as the answer, pointed the other way in time.

That one change was the largest single jump in the whole project. Measured the same way for both feature sets, it roughly doubled the signal, and the long/short Sharpe with it, from about 1.2 to 2.0. (For the quants: that is the early read on overlapping weekly samples; a stricter accounting on non-overlapping periods puts the extended signal nearer a mean IC of +0.034 and a gross Sharpe around 0.8.) What stuck with me was where the jump came from. We added no data to get it: same banks, same four years, same model. The extra signal came entirely from describing the target better: features that matched what we were asking the model to predict.

Mean IC for baseline price features versus the target-shaped residual features

Every prediction is just an API call, and those calls fan out: the baseline set, the target-shaped set, and a dozen more ideas can all be in flight at once, nothing queuing for a GPU, nothing waiting on a fit. A full four-year backtest comes back in about a minute, so the residual-feature idea was confirmed in roughly the time it took to write it. Testing a feature used to be something you set going and left; now the afternoon fills with ideas to try, and the waiting is gone.

Lever two: which rows you show it

The second lever is the context, the examples you hand the model, and it comes with its own inherited habit. With a tree, you tend to gather everything: more rows, more features, more history, all compressed into a single training set. With a foundation model, the context can stay local. You don't need to aggregate the whole world into features upfront; you can give the model the small set of examples that matter for the prediction in front of you.

We put this to the test on flight delays, predicting whether a flight would land more than fifteen minutes late across twenty US airports, from giant hubs like Atlanta and O'Hare down to quiet ones like St Louis and Pittsburgh. The tree-world approach is to train one model on all twenty at once, millions of flights. We did the opposite: we gave Seldon, for each airport, a context of only that airport's own past flights, a few thousand of them.

At the small, low-traffic airports like St Louis and Pittsburgh, the airport-specific context beat the single pooled tree where it counts. The tree took raw accuracy, which is easy when only about a fifth of flights are late and guessing "on time" is rarely wrong. But on F1, the score that rewards actually catching the delays, Seldon on a few thousand of the airport's own flights came out ahead, roughly 0.42 to 0.40, and led at four of the five. A few thousand relevant rows beat millions of indiscriminate ones on the thing you built the model for.

F1 on delayed flights at St Louis, Pittsburgh and Kansas City: airport-specific Seldon versus one pooled tree.

It is the same lesson as the features, one level up: with an in-context model you get further by curating what you show it than by piling on more, a clean break from the reflex to train on everything you have.

The other half of the loop

So far this has been about one half of the loop. Seldon makes testing a feature fast: there is no training, so each try is an API call. That fast test is what makes the next move possible: handing the whole loop to a language model.

Because a test is only an API call, there is no infrastructure to babysit, no fitting to wait on, nothing that needs a person to press go. So the model can run the experiment end to end. Give it the target and the data, and it will propose a feature, write the code, call Seldon, read the score, and decide what to try next, round after round, each round a matter of seconds. Shaping features to the target, choosing which rows to show, the two levers this whole piece is about, are both things it can work on its own. The experimentation that used to fill your week becomes something you set going and check on.

What is left for you is the part that was always the real job: choosing the problem, saying what a good answer looks like, and judging whether the features it turns up actually make sense. You are steering the search rather than turning the crank.

If you want to try this on your own data, the Seldon agent skill wraps the whole loop up for Claude Code. Point it at a CSV and the column you want predicted, and it proposes features, tests them against the model, logs what it tried and what changed, and iterates. You bring the problem and a Neuralk API key; it runs the search.

If you want to see what it looks like in action, and gets hands on, you can watch the webinar we have done on this topic.

What is left to do

Strip out the training and the tuning, and what remains is oddly close to the actual thinking: which examples represent the problem honestly; what the target really is, and what a feature shaped like it would look like; whether the column you believe you fed the model is the one it actually saw. That was always the interesting part of the work, and now there is room to spend real time on it.

The hours that used to disappear into training runs and hyperparameter sweeps come back, and they go straight into ideas: what might predict what, tried and answered in about the time it takes to read the result. For a long while the ceiling on this kind of work was how many ideas you could afford to test in a day. That ceiling has just gone up, by a lot, and that is a good problem to have.

‍