OpenAI interview question

Write a web crawler. Emphasis on multithreading/multiprocessing