I had approximate the same problem. My OS4.1FE is not run by a HD. But I SWAP not deleted and reinstalled.
In my case an old version of SLB was the problem. After I installed the version "slb_v2,29036 bytes Ver.1.19" from the CD with Partition Wizard on the HD, everything went well.
in addition I have deactivated "a1 floppy.device.kmod" in the kick layout.
A forum for AmigaOne XE and MicroA1-C specific issues.