On my main monitor, I have 8 splits:
[2.1][2.3][2.5][2.7]
[2.2][2.4][2.6][2.8]

The real windows taskbar is in [2.6] (set via Advanced Settings > Windows Taskbar Split Monitor ID).
The taskbar in [2.1] doesn't show any apps that are in it. Instead, those apps show in the main taskbar in [2.6]. These affects all apps such as firefox and notepad++.

I found the cause: Button Alignment was set to Disabled. I don't remember changing it but it's also not an obvious place to look to enable/disable buttons. Thanks!