Note that smart card usage in terminal sessions is limited to cards connected to client workstation. Smart cards connected to terminal server cannot be used in terminal session. This is limitation of terminal service.
In some cases, you have to install smart card drivers both to server and client. Mind that option of card local resource usage must be enabled on the terminal client.
The smart card itself has to be connected to client workstation, since it is the method of user personal authentication.