After some investigation and try, I have figured out what's wrong for my pn532 breakout from elechouse, here's the lib fixed:
https://github.com/picospuch/PN532

In my case, there're no additional lines other than the vcc, gnd, sdl, scl used.