Databricks interview question

How would you implement a web-crawler