Scrape exec compensation for S&P 500 companies from the SEC.

[aravindkarnam]

This challenge is inspired by this X/twitter post by Deedy.

Challenge:

Scrape exec compensation for S&P 500 companies from the SEC website

Rules:

1. You'll have to use Crawl4AI to solve this challenge.
2. You have to share you code snippets in a notebook format. (preferable in google colab)
3. The final dataset should be in CSV format. Details should include name of exec, name of the company, compensation (the more the merrier 😅)

Bonus points:

If you can scrape details of ~10000 public companies(instead of just s&p 500) as mentioned in tweet

Winners will be decided on FCFS basis.

[Umpire2018]

Fair Access
To ensure that everyone has equitable access to SEC EDGAR content, please use efficient scripting, downloading only what you need and please moderate requests to minimize server load. Current guidelines limit each user to a total of no more than 10 requests per second, regardless of the number of machines used to submit requests.
To ensure that SEC.gov remains available to all users, we reserve the right to block IP addresses that submit excessive requests. The SEC does not allow "unclassified" bots or automated tools to crawl the site. Any request that has been identified as part of an unclassified bot or an automated tool outside of the acceptable policy will be managed to ensure fair access for all users. See the Developer FAQ on how to comply with SEC's Web Site Privacy and Security Policy.

This challenge sounds fun! But I noticed that SEC's rate-limiting rules are pretty strict. I tried following the 10-requests-per-second rule and even waited 10 minutes before retrying, but I still got hit with the SEC.gov | Request Rate Threshold Exceeded message. Maybe it’s something to do with IP or concurrent requests. Either way, I think it’s best to follow the website’s guidelines—don’t want to risk getting blocked.

[aravindkarnam]

@Umpire2018 Cool. Nice to know that you are trying the challenge. Keep us posted on this thread, how it's going.