You are 1000% percent right about this caveat. And more so.
I did not write about accessibility in this article. Related, my advice on accessibility is often two parted:
1) Make sure the palette has been designed with accessibility in mind.
2) Make sure each visual is accessible - using an accessibility assessment tool.
Just because the palette is accessible, does not automatically mean all visuals will be accessible too.