On: Google Search is now less than half of Alphabet’s revenue

You can't legally make a crawler to compete with Google anymore, it's explicitly stated in a lot of robots.txt's.