FPU Context Switch
7 Bugs Between Enabling the FPU and Two Float Tasks Running in Parallel
— FPU Background —
Cortex-M7 FPU — what the hardware actually does
The Cortex-M7 implements the FPv5-D16 floating-point unit — 32 single-precision registers (S0–S31), addressable as 16 double-precision registers (D0–D15). Unlike the integer core registers, FPU registers are not automatically saved on every exception entry. The hardware uses a mechanism called lazy stacking to reduce exception latency.
CPACR — Coprocessor Access Control Register
FPU access is controlled by CPACR at 0xE000ED88. Bits [23:20] control CP10 and CP11 (the two coprocessor slots the FPU occupies). Both must be set to 0b11 (full access) before any floating-point instruction is executed. Without this, any FPU instruction causes a NOCP (No Coprocessor) UsageFault.
/* ── 3.5. FPU enable — CPACR CP10/CP11 full access ── */ ldr r0, =0xE000ED88 /* CPACR */ ldr r1, [r0] orr r1, r1, #(0xF << 20) /* CP10=11, CP11=11 */ str r1, [r0] dsb isb
This goes in the startup assembly file, after cache initialization and before the BSS zero loop. The dsb/isb pair ensures the write is committed before the next instruction fetch, which matters if the very next instruction uses the FPU.
Lazy stacking — what it is and why it matters for RTOS
Here is what lazy stacking actually does: when an exception arrives and the FPU has been used (CONTROL.FPCA = 1), the hardware reserves 18 words of stack space for the FPU state but does not immediately write the register values. The space is there, but it is empty. If the exception handler never executes a float instruction, those 18 words are never filled — the handler exits faster. Only when a float instruction actually runs inside the handler does the hardware go back and write S0–S15 and FPSCR into the reserved space. This is the "lazy" part: write it later, only if needed.
For an RTOS this matters because PendSV (the context switch handler) needs to know whether the task being switched out was using the FPU. The answer is encoded in EXC_RETURN bit[4]: 0 means FPU state is present in the frame, 1 means it is not. PendSV checks this bit to decide whether to also save S16–S31 (the upper half, which the hardware never saves automatically). This is controlled by FPCCR.LSPEN (bit 30) at 0xE000EF34, which defaults to 1.
| EXC_RETURN value | Meaning | Stack frame size |
|---|---|---|
0xFFFFFFF1 | Handler mode, MSP, no FPU | 32 bytes (8 words) |
0xFFFFFFF9 | Thread mode, MSP, no FPU | 32 bytes (8 words) |
0xFFFFFFFD | Thread mode, PSP, no FPU | 32 bytes (8 words) |
0xFFFFFFE1 | Handler mode, MSP, FPU active | 104 bytes (26 words) |
0xFFFFFFE9 | Thread mode, MSP, FPU active | 104 bytes (26 words) |
0xFFFFFFED | Thread mode, PSP, FPU active | 104 bytes (26 words) |
The key bit is bit[4] of EXC_RETURN: 1 means no FPU state in the frame, 0 means FPU state present. PendSV uses this to decide whether to save/restore S16–S31.
EXC_RETURN — the register that carries context state
When an exception is entered, the CPU writes a special value into LR called EXC_RETURN. This is not a real address — values in the 0xFFFFFFE_ and 0xFFFFFFF_ range are reserved as EXC_RETURN markers. The value encodes where to return: which mode, which stack pointer, and whether FPU state was saved.
For an RTOS, EXC_RETURN must be stored per-task. When task A uses floats and task B does not, their EXC_RETURN values differ. PendSV saves the current LR (which is EXC_RETURN on exception entry) alongside R4–R11 in the software frame, so each task restores the correct EXC_RETURN on context switch.
PendSV_Handler:
mrs r0, psp
isb
ldr r3, =g_current_task
ldr r2, [r3]
cbz r2, PendSV_restore
tst lr, #0x10 /* bit4=0 → FPU state present */
it eq
vstmdbeq r0!, {s16-s31} /* save high FPU regs if used */
stmdb r0!, {r4-r11, lr} /* save integer regs + EXC_RETURN */
str r0, [r2, #0] /* task->sp = r0 */
PendSV_restore:
ldr r3, =g_next_task
ldr r2, [r3]
ldr r3, =g_current_task
str r2, [r3]
ldr r0, [r2, #0]
ldmia r0!, {r4-r11, lr} /* restore integer regs + EXC_RETURN */
tst lr, #0x10
it eq
vldmiaeq r0!, {s16-s31} /* restore high FPU regs if used */
msr psp, r0
isb
bx lr /* EXC_RETURN → correct mode/stack/FPU */
— Debug Timeline —
How the FPU debug unfolded
arm-none-eabi-nm showed no task_float_a symbol. Root cause: -ffunction-sections in combination with function pointer references that the linker couldn't trace.bx lr with EXC_RETURN in LR from a non-exception context. This causes an INVSTATE UsageFault. Only valid inside an exception handler.-mfloat-abi=soft while our Makefile used hard-float. The debug binary was different from the flash binary. Abandoned CubeIDE, went back to pyocd + GDB..word Default_Handler for SVCall slot instead of .word SVC_Handler.— Root Causes —
Bug 1 — NOCP fault: CPACR written too late
The FPU enable block was originally placed after the BSS zero loop in startup.s. The hard-float ABI causes the compiler to emit FPU instructions for floating-point operations anywhere — including inside startup helper functions that run before main(). With CPACR still at reset value (CP10/CP11 = 0b00, access denied), the first FPU instruction causes a NOCP UsageFault.
CFSR bit 19 (NOCP) at 0xE000ED28 confirmed this. The fault address in BFAR was pointing at startup code, not at task code.
ldr r0, =0xE000ED88
ldr r1, [r0]
orr r1, r1, #(0xF << 20) /* CP10=11, CP11=11 */
str r1, [r0]
dsb
isb
0xE000ED28 = 0x00080000
bit 19 = NOCP (No Coprocessor)
→ FPU instruction executed with CP10/CP11 access denied
→ Fix: set CPACR bits [23:20] = 0b1111 in startup
Bug 2 — -ffunction-sections drops task functions
-ffunction-sections places each function in its own ELF section (.text.task_float_a etc.). The linker then removes sections with no incoming references — unless --gc-sections is explicitly used. But even without --gc-sections, sched_task_create(task_float_a, ...) passes a function pointer, and the linker sometimes cannot trace this reference through indirect calls, especially with link-time optimization or when the pointer is passed through a void* parameter.
The symptom was invisible at compile time — no warnings. arm-none-eabi-nm tamgaos_stm32h753zi.elf | grep task_float returned nothing. The binary was 5KB smaller than expected. Task stack init wrote 0x00000000 as the PC because the function didn't exist in the binary.
-ffunction-sections and -fdata-sections from CFLAGS.Verify with: arm-none-eabi-nm tamgaos_stm32h753zi.elf | grep task_float
Functions must appear with T (text) type before flashing.
Bug 3 — PendSV software frame layout mismatch
The FPU-aware PendSV handler saves context in this order (stack grows down):
vstmdb r0!, {s16-s31} — S16–S31 pushed first (highest address in software frame)
stmdb r0!, {r4-r11, lr} — R4–R11 + EXC_RETURN pushed second (lowest address, SP points here)
The restore mirrors this exactly: ldmia reads R4–R11+LR first (from low address), then vldmia reads S16–S31. The issue was that task_stack_init was initializing the FPU slots in the wrong position — placing them below R4 instead of above LR. When PendSV restored the first task, it read garbage into R4–R11 and loaded 0x00000000 into LR (EXC_RETURN).
bx lr with LR=0 → branch to address 0x00000000 → IACCVIOL (Instruction Access Violation).
Stack grows down, so push order is top-to-bottom (-- operator):
*(--sp) = xPSR; /* highest addr */
*(--sp) = PC;
*(--sp) = LR (0xFFFFFFFD);
*(--sp) = R12, R3, R2, R1, R0; /* exception frame */
*(--sp) = EXC_RETURN (0xFFFFFFFD); /* software frame start */
*(--sp) = R11 ... R4; /* lowest addr — SP points here */
/* Stack grows down. Last *(--sp) = lowest address = SP value returned. */ /* Exception frame (hardware auto-save on exception entry) */ *(--sp) = 0x01000000UL; /* xPSR — Thumb bit */ *(--sp) = (uint32_t)func | 1U; /* PC (Thumb) */ *(--sp) = 0xFFFFFFFDUL; /* LR — EXC_RETURN, Thread/PSP, no FPU */ *(--sp) = 0U; /* R12 */ *(--sp) = 0U; /* R3 */ *(--sp) = 0U; /* R2 */ *(--sp) = 0U; /* R1 */ *(--sp) = 0U; /* R0 */ /* Software frame (PendSV saves/restores this) */ *(--sp) = 0xFFFFFFFDUL; /* EXC_RETURN ← PendSV stores LR here */ *(--sp) = 0U; /* R11 */ *(--sp) = 0U; /* R10 */ *(--sp) = 0U; /* R9 */ *(--sp) = 0U; /* R8 */ *(--sp) = 0U; /* R7 */ *(--sp) = 0U; /* R6 */ *(--sp) = 0U; /* R5 */ *(--sp) = 0U; /* R4 ← SP points here */ return sp;
Bug 4 — bx lr with EXC_RETURN from Thread mode
The updated sched_start_asm ended with bx lr after loading EXC_RETURN (0xFFFFFFFD) into LR from the task software frame. This looked correct — PendSV does the same thing. But PendSV executes from Handler mode. sched_start_asm is called as a normal C function from sched_start(), which runs in Thread mode.
Executing bx 0xFFFFFFFD from Thread mode is an invalid state transition — the CPU is not in an exception handler, so there's no exception frame to unstack. The result is an INVSTATE (Invalid State) UsageFault. CFSR bit 16 (INVSTATE) confirmed this.
The fix required getting into Handler mode first, then doing the exception return. This is exactly what FreeRTOS does: use an SVC instruction to enter Handler mode, then perform the context restore inside the SVC handler.
sched_start_asm triggers SVC 0 → CPU enters Handler mode → SVC_Handler restores context → bx r14 is now a valid EXC_RETURN from Handler mode.
Bug 5 — SVC_Handler missing from vector table
After implementing the SVC approach, the svc 0 instruction executed correctly (confirmed in GDB by stepping through sched_start_asm), but instead of entering SVC_Handler, execution jumped to Default_Handler and looped forever.
The startup.s vector table had: .word Default_Handler /* 11 SVCall */. The SVCall exception slot (exception #11, vector table offset 44) pointed to the generic handler instead of our custom SVC_Handler. Even though SVC_Handler existed in the binary (confirmed with arm-none-eabi-nm), the vector table didn't know about it.
This is a common mistake when porting between targets — the vector table in startup.s must be manually updated for any new exception handler.
/* BEFORE */
.word Default_Handler /* 11 SVCall */
/* AFTER */
.word SVC_Handler /* 11 SVCall */
sched_start_asm:
cpsid i
cpsie i
cpsie f
dsb
isb
svc 0 ← triggers SVCall exception → Handler mode
nop
SVC_Handler:
ldr r3, =g_current_task
ldr r1, [r3]
ldr r0, [r1] /* r0 = task->sp */
ldmia r0!, {r4-r11, r14} /* restore R4-R11 + EXC_RETURN into LR */
tst r14, #0x10
it eq
vldmiaeq r0!, {s16-s31}
msr psp, r0
isb
mov r0, #0
msr basepri, r0
bx r14 ← valid EXC_RETURN from Handler mode ✓
Bug 6 — task_stack_init: stack was initialized but memory read as zero
After fixing the vector table, SVC_Handler was reached. Stepping through with GDB, ldmia r0!, {r4-r11, r14} executed but R14 (LR) contained 0x00000000 instead of 0xFFFFFFFD. A memory dump of the task SP address showed 17 words of zeros.
The issue was subtle: task_stack_init was being called correctly and was writing the correct values — verified by adding a breakpoint at return sp and inspecting memory immediately. But by the time SVC_Handler ran, the memory was zeroed again.
Root cause: the BSS zero loop in startup.s ran after sched_task_create was called. This can't happen in normal execution (startup → main → sched_task_create). But under certain debug scenarios or if the startup assembly was restructured, the zero loop could wipe out task stacks that were initialized before startup completed.
In practice, the real cause was a startup.s edit that accidentally moved the BSS zero loop to the wrong position. After verifying the startup sequence was correct, the task stacks initialized properly and persisted to runtime.
1. ExitRun0Mode (STM32H7 specific)
2. Cache invalidate + enable
3. FPU enable (CPACR)
4. .data copy Flash→DTCM
5. BSS zero loop ← must be here, not after
6. bl main
Never initialize task stacks before startup completes. sched_init() and sched_task_create() must be called from main(), after BSS has been zeroed.
Bug 7 — LDREX/STREX exclusive monitor cleared by lazy FPU stacking
After the scheduler was working, adding mutex_lock() in tasks that also used floats caused crashes. The pattern was consistent: mutex + FPU = board freeze. mutex alone worked, FPU alone worked.
The root cause is documented in the ARMv7-M Architecture Reference Manual (section on exclusive monitors) and in the Cortex-M7 Generic User Guide section on lazy stacking:
Lazy stacking clears the exclusive monitor. When FPCCR.LSPEN=1 (default), a SysTick interrupt arriving between LDREX and STREX can trigger lazy FPU stacking. The lazy stacking operation internally performs a memory access that clears the local exclusive monitor. The subsequent STREX sees a cleared monitor and returns 1 (failure). If the code doesn't retry on STREX failure, the mutex appears acquired but wasn't — or the retry loop spins forever preventing scheduling.
The original mutex_try_acquire called STREX once without a retry loop. When STREX failed silently, the function returned "acquired" incorrectly (because it checked store_result == 0 which happened to be true for an uninitialized stack variable), leading to two tasks both thinking they held the mutex.
PRIMASK disables all interrupts (including SysTick) for the duration of the acquire, eliminating the exclusive monitor interaction entirely.
static int mutex_try_acquire(mutex_t *m) {
uint32_t p = sched_critical_enter(); /* cpsid i */
int result = 0;
if (m->task == NULL) {
m->task = g_current_task;
result = 1;
}
sched_critical_exit(p); /* restore PRIMASK */
return result;
}
Why not disable lazy stacking instead? Setting FPCCR.LSPEN=0 forces the hardware to always save FPU state on exception entry, eliminating the race. But this adds 18 words (72 bytes) to every exception entry when FPU registers are active — increasing interrupt latency. For a preemptive RTOS at 480 MHz this is acceptable, but PRIMASK for mutex acquire is simpler and has no latency impact on the interrupt path.
— Debug Tools —
The STM32CubeIDE detour — why it didn't work
With the task_stack_init mystery unsolved (GDB showed zeros but nm showed the function existed), the temptation was to use STM32CubeIDE's step debugger to watch task_stack_init write its values in real time. The IDE has a memory browser, live variable watch, and register view — exactly what was needed.
It didn't work. The problem was the compile flags.
/* STM32CubeIDE default */ -mcpu=cortex-m7 -mthumb -mfloat-abi=soft --specs=nano.specs /* Our Makefile */ -mcpu=cortex-m7 -mthumb -mfpu=fpv5-d16 -mfloat-abi=hard
The ABI mismatch is fatal for debugging. With -mfloat-abi=soft, the IDE compiled a completely different binary — float arguments passed in integer registers, no FPU instructions generated, different calling convention. Stepping through CubeIDE's binary would show completely different behavior from the actual flashed firmware.
CubeIDE can be configured to use hard-float ABI (Project Properties → C/C++ Build → Settings → MCU GCC Compiler → General), but getting it to match the Makefile exactly — including all the kernel assembly files and section placements — would require recreating the entire build system inside the IDE. Not worth it for a debugging session.
The real answer: pyocd + arm-none-eabi-gdb using the exact same ELF produced by the Makefile. No ABI mismatch, no build system discrepancy. The binary flashed and the binary debugged are identical.
GDB Session 1 — Identifying the NOCP fault
GDB Session 2 — tracing the zero stack
GDB Session 3 — verifying SVC → SVC_Handler path
GDB Session 4 — verifying ldmia and bx r14
— Final State —
What works now
✓ FPU enabled — CPACR CP10/CP11 full access in startup.s
✓ Hard-float ABI — -mfpu=fpv5-d16 -mfloat-abi=hard
✓ PendSV FPU context switch — S16-S31 saved/restored via tst lr, #0x10
✓ EXC_RETURN stored per-task in software frame
✓ SVC-based first task launch — FreeRTOS style, valid from Handler mode
✓ FPU state isolated — task_float_a accumulates up, task_float_b decrements down, no cross-contamination
✓ Mutex fixed — PRIMASK instead of LDREX/STREX, no lazy stacking interaction
| Component | File | Status |
|---|---|---|
| CPACR enable | startup_stm32h753zi.s | Fixed — before BSS loop |
| PendSV FPU save | pendsv_handler.s | Working — S16-S31, EXC_RETURN |
| task_stack_init | scheduler.c | Fixed — software frame layout correct |
| SVC first task launch | sched_start.s | Working — FreeRTOS style |
| Vector table SVCall | startup_stm32h753zi.s | Fixed — SVC_Handler in slot 11 |
| mutex_try_acquire | mutex.c | Fixed — PRIMASK, no LDREX/STREX |
| -ffunction-sections | Makefile | Removed — task symbols stay in binary |
| uart_printf + FPU | uart.c | va_list + hard-float ABI stack usage — print after delay as workaround |
Lessons worth writing down
EXC_RETURN is a Handler mode concept
This is the single most important thing to understand about Cortex-M exception handling. 0xFFFFFFFD in LR means "return to Thread/PSP" only when the CPU is currently in Handler mode. From Thread mode, it's just an address in reserved memory. The first task must be launched via an exception (SVC is the standard choice) so that the CPU is in Handler mode when bx lr executes.
-ffunction-sections and function pointers don't mix
-ffunction-sections is useful for reducing binary size by allowing the linker to discard unused functions. But when you pass a function pointer to another function (like sched_task_create(task_led, PRIORITY_NORMAL)), the linker sees the reference through a void pointer and may not count it as a "use" of the function. The function disappears silently. Always verify with arm-none-eabi-nm that your task functions appear in the binary before flashing.
LDREX/STREX and lazy FPU stacking are incompatible without a retry loop
The ARM architecture manual documents that exclusive monitor state is not preserved across exception boundaries. With lazy stacking enabled, a SysTick interrupt between LDREX and STREX can clear the monitor via the stacking mechanism. The correct LDREX/STREX pattern is a retry loop — if STREX returns 1 (fail), the whole LDREX/STREX sequence must repeat. For a mutex, PRIMASK-based atomic access is simpler and avoids the interaction entirely.
Debug with the same binary you flash
STM32CubeIDE uses -mfloat-abi=soft by default while hard-float projects need -mfloat-abi=hard. The two ABIs generate completely different code. Debugging a soft-float build while investigating a hard-float bug produces misleading results. Always use the same ELF for flashing and debugging — pyocd + GDB with your own Makefile binary is the safe choice.
The map file and nm are your first debuggers
Before touching GDB, arm-none-eabi-nm tamgaos_stm32h753zi.elf | grep task_float immediately showed whether the function was in the binary. The map file showed the vector table size (.vectors 0x298) and whether SVC_Handler appeared at offset 44. Five minutes reading the map saved hours of GDB stepping.
Startup sequence order matters for FPU
CPACR must be set before any C code runs — including startup helper functions that the compiler might optimize to use FPU instructions. Cache initialization, BSS zeroing with memset, even simple struct initialization can generate FPU instructions under hard-float ABI. The only safe place for CPACR enable is in the startup assembly, as early as possible, before calling any C functions.
References
ARM architecture
- ARMv7-M Architecture Reference Manual (DDI0403)
- Cortex-M7 Devices Generic User Guide (DUI0646) — Lazy stacking section
- Cortex-M7 Technical Reference Manual (DDI0489)