I am significantly less worried about the tokens because it looks like most (important) 2 factor authentication keys are just regular devices without physical connectivity such as RSA SecurID 700.
After all, the second factor in 2 factor authentication with tokens means "something you have", so if you lose your device (or leave it connected for long time without reason) you already compromise your security anyway.
What I am really worried about is the statement about possible attacks on HSM devices which are permanently connected to server side systems and can be compromised using malware such as a worm or trojan with the payload crafted to crack the HSM keys and compromise the host software:
"Hardware Security Modules are widely used in banking and similar sectors where a large amount of cryptographic processing has to be done securely at high speed (verifying PIN numbers, signing transactions, etc.). A typical HSM retails for around 20 000 Euros hence is unfortunately too expensive for our laboratory budget. HSMs process RSA operations at considerable speed: over 1000 decryptions per second for 1024 bit keys. Even in the case of the FFF oracle, which requires 12 000 000 queries, this would result in a median attack time of 12 000 seconds, or just over three hours.
We hope to be able to give details of HSM testing soon."