Can you use a little smokeless powder in a cap and ball gun for more power in 1851 colt or naa companion?

people say it will be to powerful but the colt 1851 could take a 22lr powder charge and naa make a 22lr and mag the same size of the naa black powder one i got with the same metel how can it hurt