So what are Robots, Spiders, Web Crawlers, Worms, Ants?

They're all names for the same sort of thing, with slightly different connotations:

Robots
the generic name, see above.
Spiders
same as robots, but sounds cooler in the press.
Worms
same as robots, although technically a worm is a replicating program, unlike a robot.
Web crawlers
same as robots, but note WebCrawler is a specific robot
WebAnts
distributed cooperating robots.