Why your SAP ImpEx job runs all day

If a SAP Commerce ImpEx job runs for a full day while the same file imports by hand in minutes, the problem is almost never the data. It is the batch path. Scheduled and hotfolder imports serialize every row through interceptors, validation, indexing and cache eviction, single-threaded by default, while a hotfolder failure quietly retries across three cron cycles before anyone is told. This is a batch-culture problem, not a volume one.

Why does the same file take five minutes by hand and a day as a job?

Start with the case that made this a genre. A community thread on SAP's own forum describes a roughly 14,000-line ImpEx that, run as a scheduled job, kept

running for an entire day

while the identical content loaded in around 5 minutes through the Administration Console. Same file, same platform, two orders of magnitude apart. When you see that gap, stop blaming the CSV.

Three things explain most of it. First, threading: classic ImpEx runs in legacy single-threaded mode, applying every row in sequence on one worker. The distributed import in later SAP Commerce releases can spread rows across workers, but plenty of hybris-era jobs never had it switched on. Second, the write path: each row is a full service-layer save, so interceptors fire for validation, uniqueness and defaulting, the type system resolves header references, and every persisted item evicts caches and, for products, queues a Solr index update. Fourteen thousand rows becomes fourteen thousand transactions plus their side effects. Third, contention: the console run happens when you press the button, usually on a quiet node, while the cron job runs on a live instance and fights real traffic for the row locks and caches it keeps invalidating. And because indexing and cache eviction ride inside the same write, the platform can spend as much effort keeping itself consistent as it does reading your rows.

None of this shows up in the file. That is the trap. An ImpEx that is fast interactively and slow as a job is telling you the job is doing extra work per row, not that the data grew.

What happens when a hotfolder file fails?

Hotfolder is the file-drop variant: land a CSV in a watched directory, let a Spring Integration flow map it to ImpEx and import it in batches. It is convenient right up to the moment something breaks, because failure is not a single clean event. As documented in SAP KBA 3540488 (as of October 2025), a problem file is dragged through

three sequential cron jobs before marking it as a failure

Read that as an operator. One bad price file does not fail; it is retried across three separate cron passes, so the runtime you observe can be triple the real cost, the eventual error points at the last pass rather than the first cause, and the gap between "file arrived" and "someone is paged" stretches to whatever three cron intervals add up to. By the time the failure is logged, three schedules have touched the file, and the operator inherits a stack trace from the last attempt with no memory of the first. It is the same silent-batch shape we picked apart in 500 seconds and no log: batch jobs in F&O — the platform is behaving exactly as designed, which is why nobody notices until the numbers are already wrong.

Why is your master data riding the most fragile pipeline you own?

Here is the part that should worry a technology leader more than any runtime. In hybris-era stacks, catalog updates, price lists and stock corrections all move through this same ImpEx-and-hotfolder machinery. The pipeline that can run for a day, retry itself three times and report the wrong root cause is the one carrying what a product costs and whether it is available. When it stalls, a promotional price lands late; when it half-fails on a retry, a catalog version is partially synced and the storefront serves yesterday's assortment.

The same file also behaves differently depending on how it is run, which is why "it works in the console" is never proof it works in production.

Two things make this worse in the field. The header, converter and batch configuration that governs a hotfolder is specialist territory, so every change queues behind whoever owns it — the same consultant-dependency tax we described in the trade-promo module only consultants can touch. And teams that try to escape file batch by bolting an integration layer on top usually recreate the coupling one tier up, which is the exact failure mode in the ESB pattern that follows you to Kafka.

Where the fix actually lives

The way out is not a faster ImpEx; it is retiring the assumption that master data has to arrive as a file at all. Price, stock and catalog changes are per-record events, and they belong on an idempotent stream where each record is acknowledged or dead-lettered on its own, with distributed import reserved for genuine bulk. On a shared event bus, a bad price is one visible reject, not a day-long job that retries three times and pages someone at midnight. That is a re-platforming decision rather than a tuning flag, which is exactly why it keeps getting deferred.

The uncomfortable trajectory is that these pipelines never fail loudly enough to force the decision. They run a little longer each quarter as catalogs grow, absorb one more retry loop, and stay just inside the maintenance window until the day a price file lands after the promotion is already live. The brewers who move first will not be the ones whose ImpEx jobs got slowest; they will be the ones who decided that what a case of beer costs is too important to drop in a folder and hope through three cron jobs.