— Registers & Assembly Reference —
Quick Instruction Reference
On this post will be a little longer I believe — the Registers logic is finally okay — so I'm splitting it into sections:
1) LIFECYCLE OF RTOS,
2) REGISTERS,
3) ASSEMBLY,
4) DEBUG PROCESS.
RTOS Lifecycle & Stack Pointer Setup
When pushing context to the stack, the hardware saves eight 32-bit words: xPSR, ReturnAddress, LR (R14), R12, R3, R2, R1, and R0 — ref B1-643. If the processor implements the Floating-point extension (not implemented yet :) ), it can also push FP state or reserve space for it. See Stack alignment on exception entry, page B1-647.
MSP vs PSP
Normally MCU runs on MSP (Main Stack), but for real-time support every task needs its own special stack — so we MUST use PSP. This doesn't damage the MCU run cycle. All critical parts for MCU still run on MSP, so if any task breaks, only that task is affected; the MCU will not go HardFault etc.
For using PSP we need to set the SPSEL bit from the CONTROL Register. When MCU/board resets, SP_Main starts by default.
| Mode | Stack |
|---|---|
| Handler mode (always) | MSP — Reset_Handler, SysTick_Handler, PendSV_Handler, HardFault_Handler, other IRQs |
| Tasks | PSP :) |
PSP isolation does not prevent HardFault completely, but gives the system a recovery chance. If a task corrupts its own stack, the HardFault handler still runs on MSP safely. Inside the handler we can read IPSR to identify which task caused the fault, then mark that task as TASK_PAUSED and trigger a context switch to continue running other tasks without a full system reset.
This is exactly why RTOS tasks must use PSP, not MSP. MSP is the last line of defense.
CONTROL Register
31 3 2 1 0 ┌──────────┬──────┬──────┬──────┐ │ reserved │ FPCA │SPSEL │nPRIV │ │ 0...0 │ 0 │ 1 │ 0 │ └──────────┴──┴───┴──────┴──────┘
| Bit | Name | Description |
|---|---|---|
nPRIV | Privilege | If 0, MCU runs on privileged mode. We don't want unprivileged — msr, cpsid etc. won't work without privilege. |
SPSEL | Stack Select | MSP or PSP. If 0, MSP — starts as MSP after reset. |
FPCA | FP Active | Floating point bit. Must be 1 if using float numbers. (Currently 0, will implement later...) |
PRIMASK / FAULTMASK / BASEPRI
PRIMASK
31 1 0 ┌───────────────────┬──────┐ │ │ PM │ │ │ │ └───────────────────┴──────┘
The exception mask register, a 1-bit register. Setting PRIMASK to 1 raises the execution priority to 0.
CPSIE i is equivalent to writing 0 into PRIMASK. CPSID i is equivalent to writing 1 into PRIMASK. — ARM ref manual B5-802.
FAULTMASK
31 1 0 ┌───────────────────┬──────┐ │ │ FM │ │ │ │ └───────────────────┴──────┘
FM=1 masks all exceptions and faults except NMI. Difference from PRIMASK: it also blocks fault handlers.
BASEPRI
31 7 0 ┌─────────────────────────────┐ │ │ basepri │ │ │ │ └──────────────────┴──────────┘
Masks interrupts below a certain priority level. Entering 0 disables it; entering 1–255 blocks interrupts with priorities below that value.
xPSR — Combined Status Register
xPSR is actually the sum of 3 registers: APSR, IPSR, and EPSR — all together called xPSR. You can read it as a single 32-bit register.
31 30 29 28 27 24 15 8 7 0
┌──┬──┬──┬──┬──┬────┬───────────┬──────────┐
│N │Z │C │V │Q │ T │ reserved │ ISR_NUM │
└──┴──┴──┴──┴──┴────┴───────────┴──────────┘
▲ ▲ ▲ ▲ ▲ ▲ ▲
│ │ │ │ │ │ │
│ │ │ │ │ EPSR IPSR
└──┴──┴──┴──┘
APSR
| Field | Register | Description |
|---|---|---|
N Negative Flag | APSR | 1 if result is negative |
Z Zero Flag | APSR | 1 if result is zero |
C Carry Flag | APSR | 1 if carry |
V Overflow Flag | APSR | 1 if overflow |
Q Saturation Flag | APSR | 1 if DSP overflow |
T Thumb bit | EPSR | On ARM, always 1! |
ISR_NUM | IPSR | 0 in thread mode, exception number in handler mode |
On reset, the processor is in Thread mode and IPSR Exception Number is cleared to 0. Value 1 (reset exception) is transitory — software cannot see it as a valid IPSR exception number. (Page B1-625)
For programming: to set initial xPSR we only need to set the Thumb bit → 0x01000000UL (bit 24). All else zero.
SCS — System Control Space
┌─────────────────────────────────────────┐ │ SCS (0xE000E000) │ │ ├── SysTick (0xE000E010) │ │ ├── NVIC (0xE000E100) │ │ ├── SCB (0xE000ED00) <- we use │ │ │ ├── ICSR 0xE000ED04 │ │ │ ├── VTOR 0xE000ED08 │ │ │ ├── AIRCR 0xE000ED0C │ │ │ └── SHPR3 0xE000ED20 │ │ └── MPU (0xE000ED90) │ └─────────────────────────────────────────┘
Memory-mapped registers. No MRS/MSR needed — direct pointer access.
SCB Registers — ICSR, VTOR, AIRCR, SHPR3
ICSR — Interrupt Control and State Register
Provides software control of NMI, PendSV, and SysTick exceptions, and provides interrupt status information.
31 29 28 27 23 22 19 18 16 15 12 11 9 8 0
┌─────────┬──┬──────────┬──────────┬────────┬────────┬──────────┬─────────────┐
│ │PS│ │ VECTPENDING │reserved│ RETTOBASE│ VECTACTIVE │
│ │ET│ │ [20:12] │ │ │ │ [8:0] │
└─────────┴──┴──────────┴──────────┴────────┴────────┴──────────┴─────────────┘
▲ ▲ ▲ ▲
│ │ │ │
PENDSVSET waiting any active
(bit28) exception exception exception
number
| Bit | Name | Description |
|---|---|---|
| bit[28] | PENDSVSET | Write 1 → make PendSV pending. Read → current pending state. Normally software writes 1 here to request a context switch. |
| bit[27] | PENDSVCLR | Write 1 → remove PendSV pending status. |
| bit[26] | PENDSTSET | Write 1 → make SysTick pending. Read → current SysTick pending state. |
| bit[25] | PENDSTCLR | Write 1 → remove SysTick pending status. |
| bit[22:12] | VECTPENDING | Exception number of highest priority pending exception. 0 = no pending exception. |
| bit[11] | RETTOBASE | In Handler mode: 1 = no active exception other than current IPSR, 0 = other active exception exists. |
| bit[8:0] | VECTACTIVE | Exception number of current executing exception. 0 = Thread mode. |
VTOR — Vector Table Offset Register
31 7 6 0 ┌────────────┬────────────┐ │ TBLOFF │ │ │ [31:7] │ │ └────────────┴────────────┘
Holds the vector table address. When an interrupt arrives, ICSR and VTOR work together: ICSR gives VECTPENDING (which exception?), VTOR gives the table base, and VTOR + (exception_no × 4) gives the handler address to jump to.
VTOR + 0 → SP_main (reset value) VTOR + (1 × 4) → Reset_Handler VTOR + (14 × 4) → PendSV_Handler VTOR + (15 × 4) → SysTick_Handler
AIRCR — Application Interrupt and Reset Control Register
Not implemented yet. Working on software reset — might need this for a bootloader in a future issue.
31 16 15 11 10 8 7 3 2 1 0 ┌─────────────┬──────┬────────┬──────┬──────────────────────────────────── │ VECTKEY │rez │PRIGROUP│rez │SYSRESETREQ│VECTCLRACTIVE│VECTRESET│ │ [31:16] │ │[10:8] │ │ │ │ │ └─────────────┴──────┴────────┴──────┴───────────┴─────────────┴─────────┘
| Bit | Name | Description |
|---|---|---|
| bit[31:16] | VECTKEY | Must write 0x05FA when writing! |
| bit[10:8] | PRIGROUP | Priority grouping |
| bit[2] | SYSRESETREQ | Write 1 → system reset |
| bit[1] | VECTCLRACTIVE | Debug mode — clear active exception |
| bit[0] | VECTRESET | MCU core reset only |
SHPR3 — System Handler Priority Register 3
SHPR has 3 registers (SHPR1, SHPR2, SHPR3). We use SHPR3 for PendSV and SysTick: SHPR3[14] = PendSV priority, SHPR3[15] = SysTick priority.
31 24 23 16 15 8 7 0 ┌─────────┬──────────┬──────────┬──────────┐ │ PRI_15 │ PRI_14 │ rezerve │ rezerve │ │ SysTick │ PendSV │ │ │ └─────────┴──────────┴──────────┴──────────┘ 0xE000ED23 0xE000ED22
We give both 0xFF — the minimum priority. If SysTick had higher priority than PendSV, SysTick could preempt PendSV during a context switch and corrupt it. Same priority prevents this.
Vector Table Alignment
The vector table must be naturally aligned to a power of two whose value is ≥ (Number of Exceptions × 4), with a minimum of 128 bytes. On power-on or reset, the processor uses entry at offset 0 as the initial value for SP_main. All other entries must have bit[0] set to 1, because this bit defines EPSR.T on exception entry. (Directly from reference manual :) )
From TamgaOs.elf biber output — we can see the align part:
Nr Name Type Address Offset Size Flg Lnk Info Align ---------------------------------------------------------------------------------------------------------- 0 NULL 0x0000000000000000 0x000000 0x000000 0 0 0 1 .vectors PROGBITS 0x0000000000000000 0x010000 0x000190 A 0 0 4 2 .flashconfig PROGBITS 0x0000000000000400 0x010400 0x000010 A 0 0 4 3 .text PROGBITS 0x0000000000000410 0x010410 0x000A4C AX 0 0 4 4 .data PROGBITS 0x0000000020000000 0x020000 0x000000 WA 0 0 1 5 .bss NOBITS 0x0000000020000000 0x020000 0x0049C4 WA 0 0 4 6 .noinit NOBITS 0x00000000200049C4 0x000000 0x000000 WA 0 0 1 7 .ARM.attributes ARM_ATTR 0x0000000000000000 0x020000 0x00002F 0 0 1 8 .comment PROGBITS 0x0000000000000000 0x02002F 0x000049 MS 0 0 1 9 .debug_line PROGBITS 0x0000000000000000 0x020078 0x00013F 0 0 1 10 .debug_info PROGBITS 0x0000000000000000 0x0201B7 0x000072 0 0 1 11 .debug_abbrev PROGBITS 0x0000000000000000 0x020229 0x00003C 0 0 1 12 .debug_aranges PROGBITS 0x0000000000000000 0x020268 0x000060 0 0 8 13 .debug_str PROGBITS 0x0000000000000000 0x0202C8 0x00009F MS 0 0 1 14 .symtab SYMTAB 0x0000000000000000 0x020368 0x000AE0 15 130 4 15 .strtab STRTAB 0x0000000000000000 0x020E48 0x0003A5 0 0 1 16 .shstrtab STRTAB 0x0000000000000000 0x0211ED 0x0000A3 0 0 1
.noinit section already added to linker.ld for holding debug/crash data. Still not fully implemented — only a starting point :)
Assembly Instructions — Detailed
ARM uses UAL (Unified Assembly Language). We define .syntax unified at the start. See PART A4.2.2 in the reference manual — IMPORTANT!
LDR — Load from memory to register
LDR Rd, [Rn, #offset]
▲ ▲ ▲
│ │ └── how many bytes forward
│ └─────── base register
└──────────── result register
; r0 = task_t* pointer ldr r1, [r0, #0] ; r1 = task->sp (offset 0 = sp field) ldr r2, [r1, #24] ; r2 = PC from stack frame (6 × 4 bytes forward) ; Equivalent C: ; r1 = *((uint32_t*)r0); ; r2 = *((uint32_t*)(r1 + 24));
LDMIA — Load Multiple Increment After
LDMIA Rn!, {register list}
▲
└── ! means Rn is automatically updated (writeback)
ldmia r1!, {r4-r11}
; starting from r1 address:
r4 ← [r1+0]
r5 ← [r1+4]
r6 ← [r1+8]
r7 ← [r1+12]
r8 ← [r1+16]
r9 ← [r1+20]
r10 ← [r1+24]
r11 ← [r1+28]
r1 = r1 + 32 ; ← because of ! automatically updated!
STR — Store Register to memory
STR Rd, [Rn, #offset]
▲ ▲ ▲
│ │ └── how many bytes forward
│ └─────── base address
└──────────── value to write to memory
str r0, [r1, #-4]! ; first decrement r1, then write!
MOV — Write constant to register
MOV Rd, #value
▲ ▲
│ └── constant (immediate)
└─────── destination register
mov r0, #2 ; write 2 to r0
MRS / MSR — Move from/to Special Register
Note: These instructions will not work in Thread mode (they are ignored). They MUST run in Privileged mode. CONTROL[nPRIV] must be 0!
MRS Rd, special ; read special → Rd MSR special, Rd ; write Rd → special msr psp, r1 ; r1 → PSP (set task's stack pointer) mov r0, #2 msr control, r0 ; SPSEL=1 → switch to PSP ; Special register requires MRS/MSR — MOV won't work: mov r0, psp ; ← WRONG mrs r0, psp ; ← correct mov psp, r1 ; ← WRONG msr psp, r1 ; ← correct
ALIGNMENT — B1.5.7 Page B1-647 — see reference manual for full alignment rules on exception entry stack frames.
— Debug Log —
Part 6.1 — Debug Log
13 Bugs !!! Between Theory and a Blinking LED
Part 6 covered the theory — PendSV, stack frames, EXC_RETURN. This post covers what actually happened when I tried to run it on real hardware. Six distinct root causes, several hours of GDB sessions, and one blinking LED at the end.
The symptom was always the same: OpenSDA's red LED lit up, meaning the board had crashed. The causes were completely different each time. This is the log.
— Debug Timeline —
How it unfolded
scheduler.c to the build caused an instant HardFault before main() even printed anything.0x1FFF0000 (64 KB) and 0x20000000 (192 KB). BSS spanning the gap caused the startup zero-loop to write into an invalid address range.g_current_task started with a non-NULL garbage value, making the != NULL guard in sched_tick() useless.sched_start_asm finished setting PSP. PendSV ran, executed mrs r0, psp, got 0x0, tried to stmdb r0!, {r4-r11} — immediate BusFault.bx lr with EXC_RETURN value 0xFFFFFFFD from a normal function context. Only valid inside an exception handler. CPU treated it as an invalid branch and faulted.CONTROL & 0x02 guard in sched_tick() to only trigger PendSV when PSP is active. But PIT0_IRQHandler runs in Handler mode — CONTROL[1] is always 0 there. The guard permanently blocked PendSV.cpsid i at the start of sched_start_asm prevents PIT from firing mid-transition. Explicitly zeroing g_current_task in sched_init() makes the guard reliable regardless of BSS state.— Root Causes, One by One —
Bug 1 — Two RAM regions, one linker script
The K64F has two physically separate SRAM banks. Most linker examples you'll find online treat it as one contiguous block — which works for small programs that don't reach the boundary. Once s_tasks[TASK_MAX] pushed BSS past 64 KB, we were writing into the gap between the two banks.
The startup BSS zero-loop would appear to succeed (no fault, loop exits), but the data at addresses above 0x20000000 was never actually zeroed — it was reading back whatever the power-on reset left there.
m_data (rwx) : ORIGIN = 0x1FFF0000, LENGTH = 64K
m_data_2 (rwx) : ORIGIN = 0x20000000, LENGTH = 192K
Place BSS, data, and stack all in m_data_2. The gap disappears.
/* BEFORE — one region, spans the physical gap */ MEMORY { RAM (rwx) : ORIGIN = 0x1FFF0000, LENGTH = 256K } /* AFTER — two regions matching the silicon */ MEMORY { FLASH (rx) : ORIGIN = 0x00000000, LENGTH = 1024K m_data (rwx) : ORIGIN = 0x1FFF0000, LENGTH = 64K m_data_2 (rwx) : ORIGIN = 0x20000000, LENGTH = 192K } _stack_top = ORIGIN(m_data_2) + LENGTH(m_data_2);
Bug 2 — Vector table had 84 entries instead of 100
The hand-written startup assembly vector table stopped at IRQ67. K64F's PIT0 is IRQ48, which is slot 64 in the table (16 core exceptions + 48 IRQs). With only 84 words in the table (16 core + 68 IRQs), the slot was there — but IRQ68 through IRQ83 were missing entirely.
The map file was the giveaway: .vectors 0x00000000 0x150. That's 336 bytes = 84 words. It should be 0x190 = 400 bytes = 100 words.
.word Default_Handler entries for IRQ68–IRQ83. Verify in the map file: .vectors 0x00000000 0x190.
Bug 3 — BSS zero-loop silently failed
The startup BSS loop used blt (Branch if Less Than — signed) to compare two RAM addresses. Addresses above 0x80000000 are negative in two's complement. 0x1FFF0000 and 0x20000000 are both positive, so the loop ran. But depending on the exact BSS layout, the loop would sometimes exit immediately or run indefinitely — impossible to reproduce reliably.
The real fix was moving BSS into m_data_2 (Bug 1), but additionally: g_current_task and g_next_task are now explicitly zeroed inside sched_init(), making the scheduler correct regardless of what the startup zero-loop does or doesn't do.
void sched_init(void) {
g_current_task = (task_t *)0U; /* explicit zero — don't trust BSS */
g_next_task = (task_t *)0U;
s_task_count = 0U;
SCB_SHP_PENDSV = 0xFFU;
SCB_SHP_SYSTICK = 0xFFU;
// ... idle task init ...
}
Bug 4 — PendSV triggered before PSP was set
The sequence in sched_start_asm was: set PSP → set CONTROL → jump to task. But PIT0 was already running with a 1 ms period. Between setting CONTROL and the task actually starting, a PIT interrupt could fire. sched_tick() would call trigger_pendsv(). PendSV would run, read PSP (still partially set up or 0x0), and fault immediately.
cpsid i / cpsie i — disable interrupts for the duration of the critical setup, re-enable once the first task is safely running.
sched_start_asm:
cpsid i ; mask all interrupts
ldr r1, [r0, #0] ; r1 = task->sp
ldmia r1!, {r4-r11} ; restore software frame
msr psp, r1 ; PSP = hardware frame base
isb
mov r0, #0x02
msr control, r0 ; Thread mode, PSP
isb
cpsie i ; interrupts back on — PSP is now valid
isb
ldr r2, [r1, #24] ; PC from hardware frame (offset 6*4)
bx r2 ; jump to task entry point
Bug 5 — EXC_RETURN from a non-exception context
Early versions of sched_start_asm ended with movw lr, #0xFFFD / movt lr, #0xFFFF / bx lr. That EXC_RETURN value is only meaningful when executed from inside an exception handler. From a normal function call, the CPU doesn't recognize it as EXC_RETURN — it treats it as a branch to address 0xFFFFFFFD, which is in the reserved address space. Immediate fault.
Several workarounds were tried — SVC to get into exception context, naked functions — before landing on the simplest solution: jump directly to the task's PC and let the hardware frame do its job naturally when the task eventually returns to the OS (which it never does in a while(1), but the frame is still correct for PendSV context switching).
bx r2. PSP is already pointing at the frame, so PendSV can save/restore correctly on the next context switch.
Bug 6 — CONTROL register always reads 0 inside an IRQ
To prevent PendSV from firing when PSP wasn't set up yet, a guard was added to sched_tick():
uint32_t ctrl;
__asm volatile("mrs %0, control" : "=r"(ctrl));
if (ctrl & 0x02U) { /* always 0 in Handler mode! */
trigger_pendsv();
}
This looks reasonable — only trigger PendSV when PSP is active (CONTROL[1] = 1). But PIT0_IRQHandler runs in Handler mode. In Handler mode, the CPU always uses MSP and CONTROL[1] is always 0. The guard was permanently false. PendSV was never triggered after the first task started.
GDB caught this through a different symptom: both tasks showed state = TASK_READY and delay_ticks = 0, meaning they were ready to run but never getting scheduled. The idle task was the one running instead.
g_current_task = NULL in sched_init() plus the cpsid/cpsie in sched_start_asm. Once the first task is running, PendSV can always be triggered safely.
— GDB Sessions —
Session 1 — Catching the HardFault location
The board crashed before printing anything. First step was confirming the crash was a HardFault and finding where it happened.
The lr = 0xfffffff1 meant a fault-within-a-fault — the CPU tried to handle a fault and faulted again. Combined with psp = 0x0, the picture was clear: PendSV ran before PSP was initialized.
Session 2 — task_led never actually running
After fixing the vector table and linker script, the board stopped crashing immediately, but the LED never blinked. Breakpointing task_led to confirm it was being reached:
The memory fault on GPIO access was a red herring — the real cause was PendSV firing between sched_start_asm setting CONTROL and the task actually running, with PSP still partially initialized.
Session 3 — confirming PSP = 0x0
After adding the cpsid/cpsie guard in sched_start_asm and zeroing g_current_task in sched_init(), checking the register state at the moment of crash:
Session 4 — idle task running instead of scheduled tasks
UART was printing, LED was not blinking. Connected GDB mid-run — caught the idle task running with both real tasks READY:
This was the last bug. Removing the CONTROL & 0x02U guard from sched_tick() let PendSV fire normally, and the scheduler started switching tasks correctly.
— Final State —
What works now
✓ Two tasks running concurrently — task_led (500 ms blink) and task_uart (1 s UART print)
✓ PendSV context switch — PSP-based, R4-R11 saved/restored by software
✓ sched_delay_ms — tasks block and yield correctly
✓ Idle task — runs when all tasks are blocked, WFI for power saving
✓ Priority scheduler — lowest number wins, round-robin among equal priorities
| Component | File | Status |
|---|---|---|
| Linker script | boards/k64f/linker.ld | Fixed — two RAM regions |
| Vector table | boards/k64f/startup_k64f.s | Fixed — 100 entries |
| Scheduler core | kernel/core/scheduler.c | Working |
| Context switch | kernel/arch/cortex_m4/pendsv_handler.s | Working |
| First task launch | kernel/arch/cortex_m4/sched_start.s | Working — cpsid/cpsie guarded |
| PIT driver | drivers/k64f/pit.c | Working — calls sched_tick() |
| Mutex / semaphore | — | Implemented, testing in progress |
| MCG 120 MHz | drivers/k64f/mcg.c | Fixed — HGO0 removed, flash wait state added |
| PIT driver | drivers/k64f/pit.c | Fixed — 4x NOP clock gate barrier |
| Critical section | include/sched_critical.h | Fixed — BASEPRI replaced with cpsid/cpsie |
| Idle TCB stack | kernel/core/scheduler.c | Fixed — declaration order corrected |
— Part 6.2: Getting sched_delay_ms to actually block —
Bug 7 — HGO0 flag: oscillator would not start
After the scheduler bugs were resolved, mcg_init_120mhz() was next. The MCG C2 register was written with MCG_C2_RANGE0(2) | MCG_C2_HGO0_MASK | MCG_C2_EREFS0_MASK. The HGO0 bit selects high-gain oscillator mode for crystals that need it. The 50 MHz clock source on this board does not — it drives the MCG external reference directly. With HGO0 set, the oscillator simply never started and the PLL lock loop spun forever, hanging the system before main() printed anything.
The symptom was identical to an earlier hang: board appeared dead, no UART output, LED did not respond. Isolated by removing mcg_init_120mhz() and testing with a bare GPIO toggle loop — which worked fine. Then re-added MCG piece by piece. The hang disappeared when HGO0 was removed.
MCG_C2_HGO0_MASK from the C2 write.MCG->C2 = MCG_C2_RANGE0(2U) | MCG_C2_EREFS0_MASK;Bug 8 — Flash wait state not set before clock switch
mcg_init_120mhz() set SIM->CLKDIV1 (core /1, bus /2, flash /5) as its very first action. At that moment the CPU was still running on FEI (~20 MHz) with zero wait states. After CLKDIV1 was written the flash controller was immediately being asked to serve instructions at 120 MHz — but with zero wait states configured it can only reliably do ~25 MHz. The result was corrupt instruction fetches: random HardFaults, or code that appeared to run but produced wrong results.
K64F reference manual §3.8.2: flash requires RWSC = 5 for 120 MHz operation. FMC->PFB01CR bits [19:16] must be set before raising the clock.
FMC_PFB01CR RWSC = 5 before SIM->CLKDIV1.FMC_PFB01CR = (FMC_PFB01CR & ~(0xFU<<16U)) | (5U<<16U);Also increased crystal stabilisation delay from 10 000 to 50 000 NOPs (~2.5 ms).
Bug 9 — sched_start_asm: bx r0 instead of EXC_RETURN
The original sched_start_asm called bx lr with lr = 0xFFFFFFFD (Thread/PSP EXC_RETURN). This is only valid inside an exception handler. Called from sched_start() as a normal C function, the CPU was in Thread mode. Executing bx 0xFFFFFFFD from Thread mode is an invalid branch to reserved memory: instant UsageFault.
The fix was to read PC directly from the task hardware frame at PSP+24 and branch with bx r0. CONTROL.SPSEL is set to 1 first so the CPU switches to PSP. The hardware frame sits untouched at PSP — when PendSV context-switches away and back, it restores the rest of the frame normally via EXC_RETURN.
ldmia r1!, {r4-r11} and msr psp, r1:ldr r0, [r1, #24] // PC from hardware framecpsie ibx r0 // branch directly, not EXC_RETURNBug 10 — LR initialised to 0xFFFFFFFE in task stack frame
Cortex-M4 EXC_RETURN values: 0xFFFFFFF1 (Handler/MSP), 0xFFFFFFF9 (Thread/MSP), 0xFFFFFFFD (Thread/PSP). Everything else in the 0xFFFFFFF_ range is reserved. task_stack_init wrote 0xFFFFFFFEUL as the initial LR. Caught in disassembly: mvn r2, #1 = NOT(1) = 0xFFFFFFFE. When PendSV returned via this LR, the CPU took a UsageFault.
task_stack_init:*(--stack_top) = 0xFFFFFFFDUL; // Thread/PSP EXC_RETURNConfirmed in disassembly:
mvn r2, #2 = NOT(2) = 0xFFFFFFFD.Bug 11 — idle TCB stack overwriting s_task_count
Static variable layout in BSS follows declaration order. scheduler.c had: s_idle_tcb at 0x20000008, then s_task_count at 0x20000818, then s_tasks at 0x2000081C. A full task_t is 2064 bytes (0x810), so &s_idle_tcb.stack[512] = 0x20000818 — exactly s_task_count. task_stack_init writes the initial frame downward from that address, directly clobbering s_task_count and the start of s_tasks.
GDB caught it: psp = 0x20000818 <s_task_count> — the CPU was running the idle task with its stack pointer sitting on top of the task count variable.
static uint8_t s_task_count;static task_t s_tasks[TASK_MAX];static task_t s_idle_tcb; // stack grows into free space belowBug 12 — SIM clock gating: PIT->MCR write silently dropped
After SIM->SCGC6 |= SIM_SCGC6_PIT_MASK, pit_init immediately wrote PIT->MCR = 0x01. The write was silently discarded — the peripheral bus clock takes a few cycles to propagate after the clock gate opens. MCR remained at its reset value of 0x02 (MDIS = 1, module disabled). With MDIS set, PIT counters do not run and no interrupts fire.
This was invisible in debug builds: the debugger halt/resume overhead added enough cycles that the clock was always ready. In release builds at full speed the write was lost every time. Confirmed with pyocd: read32 0x40037000 → 0x00000002. After adding four NOPs: read32 0x40037000 → 0x00000001.
SIM->SCGC6 |= SIM_SCGC6_PIT_MASK;__asm volatile("nop"); __asm volatile("nop");__asm volatile("nop"); __asm volatile("nop");PIT->MCR = 0x00000001UL;Bug 13 — BASEPRI masking PendSV indefinitely
sched_critical_enter() wrote BASEPRI = 0x30 (priority level 3). BASEPRI blocks all exceptions with numeric priority ≥ that value. PIT0 was at priority 4 (0x40) and PendSV at 15 (0xFF) — both blocked. The critical section exited correctly, but if trigger_pendsv() fired just before sched_critical_exit(), the pending bit was set with BASEPRI still active. A subsequent re-entry into a critical section before PendSV got CPU time would block it again. In practice sched_delay_ms returned without a context switch ever happening.
sched_critical.h:__asm volatile("mrs %0, primask" : "=r"(p));__asm volatile("cpsid i");Exit:
__asm volatile("msr primask, %0" :: "r"(p));— GDB Sessions: Part 5.2 —
Session 5 — sched_delay_ms: PendSV fires but task never resumes
After oscillator and flash wait-state fixes, the scheduler appeared to run but sched_delay_ms never blocked — UART printed at full speed, LED stayed solid. Broke on sched_delay_ms and inspected state after the critical section.
Session 6 — PSP pointing at s_task_count
With BASEPRI fixed, PendSV fired but caused an immediate HardFault. Connected GDB mid-run and interrupted:
Session 7 — PIT->MCR = 0x02: module disabled in release build
After the stack overflow fix, the scheduler worked in debug builds but not release. LED stayed solid, UART printed only once. Used pyocd commander to inspect PIT registers while running:
Lessons worth writing down
Check the silicon memory map
The K64F has two physically separate SRAM banks. Every tutorial I found treated it as one block. The reference manual (Chapter 4) is the only source that gets this right. Read the manual, not the tutorial.
CONTROL register behavior changes in Handler mode
In Handler mode (any exception/IRQ), the CPU always uses MSP and CONTROL[1] reads back 0, regardless of what Thread mode is using. A guard built on reading CONTROL inside an IRQ handler does nothing.
EXC_RETURN is only valid inside an exception handler
0xFFFFFFFD in LR means "return to Thread/PSP" only when the CPU is in exception context. From a regular function call, it's just an address in reserved memory — instant fault.
GDB + pyocd: always use --persist
pyocd gdbserver -t k64f --persist keeps the server alive across disconnects. Without it, every time GDB exits you need to restart pyocd before reconnecting.
map file is your first debugger
Before touching GDB, the map file told us: vector table size wrong (0x150 vs 0x190), sched_start_asm landing before Reset_Handler (link order bug), BSS at wrong address (linker region issue). Five minutes reading the map saved hours of stepping through assembly.
Clock gating requires a barrier
On K64F (and most Kinetis devices), writing a clock gate enable bit in SIM->SCGCx does not make the peripheral instantly accessible. The bus fabric needs a few cycles to propagate the enable. Writing a peripheral register in the very next instruction will be silently dropped. The manual documents this as "two bus clock cycles" but in practice four NOPs is the safe minimum. This only shows up in release builds — debug builds always have enough overhead to hide it.
pyocd commander is faster than GDB for register hunts
For questions like "is this peripheral register what I think it is?", pyocd commander with read32 is faster than setting a breakpoint and inspecting memory in GDB. No ELF required, works while the program is running, and the output is immediate. The PIT->MCR = 0x02 bug would have taken much longer without it.
Static variable declaration order matters for stacks
The C standard does not guarantee BSS layout, but in practice GCC lays out static variables in declaration order. If a task TCB with a large embedded stack array is declared before smaller variables, its stack can silently overwrite them. The fix is mechanical — declare large stack-bearing structs last — but the bug is invisible until you look at the actual addresses with GDB or a map file.
BASEPRI is a precision instrument — use cpsid for simplicity
BASEPRI lets you block interrupts above a specific priority threshold, which is useful for leaving high-priority ISRs (like a DMA completion handler) running during a critical section. But it also means PendSV — at the lowest priority — is always blocked whenever BASEPRI is non-zero. If your critical section accidentally stays active longer than expected (nested locks, early return paths), PendSV never fires. For a simple preemptive scheduler, cpsid i / cpsie i is the right tool: simpler semantics, impossible to accidentally block PendSV.
One thing that surprised me: the blt vs blo distinction in the BSS loop — signed vs unsigned branch — is the kind of bug that appears totally benign in simulation (small programs, addresses always positive) and only shows up on real hardware with a large enough BSS to cross the 0x80000000 boundary. It's not even a K64F-specific issue; any Cortex-M device with upper-bank RAM can hit this.
— Chapter V: ELF Analysis with biber —
Reading the binary with biber
After the scheduler was running, I used biber — a custom ELF analysis tool — to inspect the final binary. The goal was to verify that everything landed where the linker script said it should: correct RAM regions, correct section sizes, correct symbol addresses. It's the kind of check that catches subtle linker issues before they become runtime bugs.
CLASS : ELF32 DATA : LittleEndian TYPE : ET_EXEC (executable) MACHINE : .ARM Entry : 0x00000411 ← Reset_Handler + Thumb bit PROGRAM HEADER Type Flg VAddr PAddr FileSize MemSize LOAD R-X 0x0000000000000000 0x0000000000000000 0x00000DE4 0x00000DE4 LOAD RW- 0x0000000020000000 0x0000000000000DE4 0x00000000 0x000049B8
Two load segments — exactly what we expect. The first is read/execute (flash), the second is read/write (RAM). The RAM segment has FileSize = 0 because it's all BSS — no initialized data to load from flash.
Section breakdown
The section header table is where the real layout lives. Each section maps to a specific region of the address space.
| Section | Address | Size | Notes |
|---|---|---|---|
| .vectors | 0x00000000 | 0x190 = 400 B | 100 entries × 4 bytes — correct |
| .flashconfig | 0x00000400 | 0x010 = 16 B | Flash security config at fixed offset |
| .text | 0x00000410 | 0x9D4 = 2516 B | All code — Reset_Handler starts here |
| .data | 0x20000000 | 0x000 = 0 B | No initialized globals — clean |
| .bss | 0x20000000 | 0x49B8 = 18.9 KB | Task stacks, globals — all in m_data_2 |
| .noinit | 0x200049B8 | 0x000 = 0 B | Reserved for crash log — empty for now |
The .vectors size was the first thing I checked. Earlier in the debug process, the table was 0x150 (84 entries) instead of 0x190 (100 entries). PIT0 is IRQ48 — slot 64 in the table — so with only 84 entries, the PIT0 handler slot was missing entirely. biber made this immediately obvious.
The BSS at 18.9 KB is dominated by the task stacks: 8 tasks × 512 words × 4 bytes = 16 KB, plus globals, ring buffers, and TCB overhead. All of it sits in m_data_2 (0x20000000, 192 KB) — the correct bank.
Symbol table — what lives where
The symbol table shows every named object in the binary with its address and size. A few entries worth highlighting:
; Core scheduler globals g_current_task 0x20000000 4B GLOBAL ← BSS start, first word in RAM g_next_task 0x20000004 4B GLOBAL ; Task storage s_idle_tcb 0x20000008 2064B LOCAL ← 1 TCB = 512 stack + overhead s_tasks 0x2000081C 16512B LOCAL ← 8 × 2064 = 16512 bytes s_task_count 0x20000818 1B LOCAL ; UART ring buffer s_tx_buf 0x200048A4 256B LOCAL s_tx_head 0x200049A4 4B LOCAL s_tx_tail 0x200049A8 4B LOCAL ; Synchronization primitives g_pit_sem 0x200049AC 8B LOCAL ← count(4) + max(4) g_uart_mutex 0x200049B4 4B LOCAL ← owner pointer ; BSS boundary _bss_end 0x200049B8 ← matches .noinit start exactly _noinit_start 0x200049B8 _stack_top 0x20030000 ← 192KB above m_data_2 base
One thing that stands out: g_current_task sits at the very first address of BSS (0x20000000). This caused a subtle boot bug — because BSS wasn't being zeroed by startup, g_current_task contained garbage. The sched_init() call now explicitly zeroes it before anything else touches the scheduler.
Key findings and what they confirmed
Entry point thumb bit
The entry point is listed as 0x00000411, but Reset_Handler is at 0x00000410. The +1 is the Thumb bit — the CPU reads this from the reset vector and sets the T bit in EPSR before fetching the first instruction. Without it, a Cortex-M4 would take a UsageFault immediately on reset. The linker adds this automatically when you use .thumb_func or compile with -mthumb.
$t and $d markers
The symbol table is full of $t and $d entries with no name. These are assembler-generated markers: $t marks the start of a Thumb code region, $d marks a literal pool (data embedded in the code section). Debuggers and disassemblers use these to switch between Thumb disassembly and hex display — without them, a disassembler would try to decode literal pool data as instructions and produce nonsense.
Link order confirmed
Earlier, sched_start.o and pendsv_handler.o were landing in .text before startup_k64f.o, pushing Reset_Handler to a higher address. The fix was moving their code into a .kernel_asm section, placed after .text in the linker script. biber confirms it worked: Reset_Handler = 0x411 (start of .text), and sched_start_asm = 0xD81, PendSV_Handler = 0xDA9 — both well after the startup code.
.data is empty — good
A non-zero .data section means the linker has to copy initialized values from flash to RAM on startup. Every global like int x = 5 adds to it. TamgaOS has none — every global is either zero-initialized (BSS) or set explicitly at runtime. This keeps the startup path simple and eliminates one class of potential bugs.
Cross-checking with arm-none-eabi-size
After biber, a quick sanity check with the standard toolchain size utility:
text data bss dec hex filename 2932 0 18872 21804 552c tamgaos_k64f.elf
The numbers match biber exactly — but the columns mean slightly different things. arm-none-eabi-size groups all flash-resident sections into text: that's .vectors (0x190) + .flashconfig (0x10) + .text (0x9D4) = 0xB74 = 2932. biber shows them separately. Both are correct — just different views of the same layout.
data = 0 confirms there are no initialized globals. bss = 18872 = 0x49B8 — matches _bss_end - _bss_start from the symbol table exactly. When these three numbers agree across two independent tools, the linker script is doing what you think it's doing.
The tool that pays for itself: reading the ELF directly — sections, symbols, program headers — takes five minutes and answers questions that GDB sessions alone never quite settle. Where did this symbol land? Is the section the right size? Did the linker script do what I thought? biber answers all three without needing the hardware.
References
Everything above traces back to these primary sources. When in doubt, read the spec — not a blog post. Including this one.
ARM architecture
- ARMv7-M Architecture Reference Manual (DDI0403E) — PDF
- ARMv7-M Architecture Reference Manual
- Cortex-M4 Devices Generic User Guide (DUI0553A)
- Cortex-M4 Processor Technical Reference Manual — Table 4-1 system control registers