The midterms were a referendum on the economy/inflation and the Iran war, which also worsened the economy. Voters would have mostly ignored his policies if things were booming and jobs were plentiful.
If the Democrats don't keep talking loudly and constantly about improving the economy, they will get kicked right back out.