Real brick walls use mortar, which adds tensile strength, and the tensile cohesion allows side load to transfer to the wider footing. This is substantially different than your model, where there is only friction holding the bricks together.

Perhaps Blender isn't the right tool for this simulation, but you use what you know I guess.

Can you figure out a way to add some tensile connection between the bricks? I bet even a basic conditional constraint like "hold together until force exceeds X threshold" would give a lot more accuracy. Mortar is quite brittle.