Why Use ScrapingBee
While we endeavor to support as many websites as possible, sometimes we are unable to scrape certain pages. This is where using a service like scrapingbee helps to bridge the gap. Scrapingbee utilizes an advanced bot avoidance system to scrape webpages and turn them into links which can then be put into Pitchlane. This allows users to still automate their outreach, even in cases where Pitchlane cannot natively offer those websites as an option.
How to Set Up ScrapingBee (Automated)
Step 1: Creating an Account
First navigate over to ScrapingBee. You can do so by clicking on this link: Scrapingbee.com
Then hit the sign up button.
Step 2: Configure the Scraper
From the dashboard, click on “HTML Request Builder”. Then input your URL. From there ensure “domcontentloaded” has been enabled under “Wait for Browser” and Window Width is set to “1920” and Window Height is set to “1080” and importantly Screenshot enabled.
Step 3: Select “Try it”
Step 4: Save the Request
Step 5: Open up your CSV in Google Sheets or Excel. Make a new column and insert the below formula. Making sure to replace the “APIKEY” with your API Key from ScrapingBee & changing “D2” to the column with your websites.
="https://app.scrapingbee.com/api/v1?api_key=yourapikey&url=" & ENCODEURL(D2 & IF(ISNUMBER(FIND("?", D2)), "&", "?") & "%2Fanything%3Fjson&screenshot=true")Step 6: Export your CSV and upload into Pitchlane and render your videos letting Pitchlane and ScrapingBee handle the rest!
Additional Notes: Depending on your request configuration you may have additional elements in your Scrapingbee request. To add these simply review the URL and see where additional arguments may exist versus the excel formula provided above, and paste them in where the “%2Fanything%3Fjson&screenshot=true” part is.
How to Set Up ScrapingBee (Manual)
Step 1: Creating an Account
First navigate over to ScrapingBee. You can do so by clicking on this link: Scrapingbee.com
Then hit the sign up button.
Step 2: Configure the Scraper
From the dashboard, click on “HTML Request Builder”. Then input your URL. From there ensure “domcontentloaded” has been enabled under “Wait for Browser” and Window Width is set to “1920” and Window Height is set to “1080” and importantly Screenshot enabled.
Step 3: Select “Try it”
Note: You can use the cURL for an API to automate this process
Step 4: Copy Image
Step 5: Navigate to imgBB and upload.
Step 6: Go to link provided and copy image address for the direct link
Step 7: Open your Pitchlane Campaign
Head on over to the Import Leads Page & Create a Single Video
Step 8: Input the direct link and render the videos
Note: This process is likely to be much faster and time effective if you automate it using their API which can be found here: https://www.scrapingbee.com/documentation/