I have set the switches for I2C mode.
Connecting to 3.3V instead of 5V didn't help.
Funny thing is if I disconnect VCC shortly while the Arduino sketch is running setup() then it works!
It skips getting the firmware version, goes into loop() mode and waits for a card.
I place a card and it detects it!
I think there is something wrong with the PN532_I2C library.
I also found that it does work with the nfc library (nfc.cpp + nfc.h) that Elechouse provides.
So I am going to use that one instead.