Make a scene analysis with the tools that you have and check for potential optimizations. As soon as you start lagging in a scene, there is something wrong (if you have a normal computer)
You cannot simply say "300 polys for this, 502 polys for that". Try to optimize as much as possible.