.avif)
Sign Up for Abstract Web Scraping API
If you are not familiar with Abstract Web Scraping API, then signup for your free Abstract account to get access to all the APIs. Once logged in, you can access the Web Scraping API from the dashboard.
You can access your API key from the Web Scraping API console.
You can make a test request from the console to scrape the URL provided by default. You should see a large blob of HTML returned by the API. This ensures that your API key works fine and you are now ready to integrate this API into the PHP application.
Demo Web Scraping App with PHP
Let's put this Web Scraping API to some use by building a demo application. PHP is one of the most widely used programming languages for building web applications, and Laravel is a popular PHP web framework. We will leverage both of these technologies to show you how to build a basic web scraping application.
Follow these steps in the rest of this post to witness how you can create a Laravel based web app for web scraping in a few steps. But first here are a few prerequisites you should have to ensure that you have the right developer environment for creating this application.
Prerequisites
- PHP Runtime: Make sure you have a PHP 8 runtime available on your developer environment
- PHP Toolchain for Laravel: Make sure you also have the composer package manager
Step 1 - Create a New Laravel Project
Open a terminal and run the following composer command to create a new Laravel project named php_abstractapi.
composer create-project --prefer-dist laravel/laravel php_abstractapi
This will create a directory named php_abstractapi under the present working directory where the command is executed. This is the project directory of this demo app containing all the boilerplate code and dependencies. Make sure to change to this directory for executing all further commands from the terminal.
Open your favorite IDE and check out the directory structure of the project directory
Step 2: Test the Default Laravel App
The empty Laravel project can be tested by launching it from the terminal.
php artisan serve
This will start a development web server that hosts the default Laravel app at https://127.0.0.1:8000. You can check out the default landing page for this app on the browser.
Step 3: Add the API Credentials for Demo App
Open the environment file for the project and add two new environment variable entries for the Abstract API URL and Abstract API key.
File: .env
ABSTRACT_API_KEY=
ABSTRACT_API_URL=
Replace the placeholder <YOUR_ABSTRACT_API_KEY> with your Abstract API key and <ABSTRACT_WEB_SCRAPING_API_ENDPOINT> with the API URL. The URL can be found in the live test console within the Abstract API console.
As of now, this URL is set to https://scrape.abstractapi.com/v1
Step 4: Add the HTTP Helper Class for Handling Abstract API
Create a helper class, AbstractAPI.php under the Http subdirectory.
File: app/Http/Helpers/AbstractAPI.php
Add the following PHP code snippet within this file:
namespace App\Http\Helpers;
use Illuminate\Support\Facades\Http;
class AbstractAPI
{
public function make_request(string $url)
{
$api_url = env('ABSTRACT_API_URL', '');
$api_key = env('ABSTRACT_API_KEY', '');
$api_url = trim($api_url);
$api_key = trim($api_key);
if( $api_url == "" || $api_key == "" )
{
return "Error Occured: ABSTRACT API configuration error.";
}
try
{
$request_url = $api_url."?api_key=".$api_key."&url=".$url;
$res = Http::withHeaders([
"Content-Type" => "application/plain-text",
])->get($url);
return $res;
}
catch (Exception $e)
{
throw new ErrorException($e->getMessage());
}
}
}
This helper class handles the call to Abstract Web Scraping API from the PHP backend.
Step 5: Add a New Controller Named WebscrapeController
From the terminal, add a new controller named WebscrapeController to the project.
php artisan make:controller WebscrapeController
This will create a new PHP file
File: app/Http/Controllers/WebscrapeController
Replace the default content of the file with the following code:
namespace App\Http\Controllers;
use Illuminate\Http\Client\ConnectionException;
use Illuminate\Http\Request;
use App\Http\Helpers\AbstractAPI;
class WebscrapeController extends Controller
{
public function index()
{
return view('webscrape');
}
public function requestapi(Request $request)
{
if ($request->isMethod('post'))
{
$url = $request->request->get('url');
if( $url == "" )
{
return response("Invalid Url", 400)->header('Content-Type', 'text/plain');
}
$url_valid = filter_var($url, FILTER_VALIDATE_URL);
if( !$url_valid )
{
return response("Bad Request", 400)->header('Content-Type', 'text/plain');
}
try{
$response = (new AbstractAPI())->make_request($url);
return $response;
}
catch(ConnectionException $e)
{
return response("Host could not be resolved", 400)->header('Content-Type', 'text/plain');
}
catch(Exception $e)
{
return response("Error : ".$e->getMessage(), 400)->header('Content-Type', 'text/plain');
}
}
else
{
return response("Invalid Method", 400)->header('Content-Type', 'text/plain');
}
}
}
This controller defines a custom API endpoint, '/requestapi'. This API accepts the URL from the frontend UI and passes it to Abstract Web Scraping API for scraping the contents of the URL. As part of handling the Abstract API call, this controller also defines ConnectionException to catch invalid URLs.
This controller also defines the home page view for the UI which is labeled as 'webscrape'.
Step 6: Update the App Routes
The app has two routes. One is '/' for displaying the home page of the frontend UI of the demo app. And the other is '/requestapi' for triggering the scraping request.
You must register these routes for the demo app in Laravel. To achieve this, replace the content of the routes definition.
File: routes/web.php
use App\Http\Controllers\WebscrapeController;
use Illuminate\Support\Facades\Route;
Route::get('/', [WebscrapeController::class,'index']);
Route::post('/requestapi', [WebscrapeController::class,'requestapi']);
Step 7: Create the HTML and JavaScript for the Demo App UI
At this point, all the backend PHP logic is built for the demo app. Now the last thing is the graphical user interface (UI) which is an HTML page.
This is a Bootstrap based HTML code for a web form that lets the user input a URL and submit the form. The JavaScript code links the form to the '/requestapi' endpoint of the PHP backend to send web scraping requests with the URL.
Step 8: Adding Bootstrap CSS to the code
To ensure that Bootstrap CSS styles are applied to the frontend UI, get the Bootstrap.min.css from the link below:
https://getbootstrap.com/docs/5.0/getting-started/download/
Create a sub-directory 'css' within the public sub-directory of the project and copy the downloaded bootstrap.min.css into it.
Alternatively, you can also add a link to the CDN source of bootstrap in the HTML file header.
With this step, we are done with all the code development for this demo app. Make sure to save all the files before proceeding with the next steps.
Step 9: Relaunch the Laravel Server
Relaunch the Laravel development server which was earlier run in step 2 to test the default Laravel app.
Now you should see the demo app UI on the browser as per the view created in step 7.
Step 10: Test the Demo App
Now you are ready to test the app.
Enter any website URL in the form and click submit. The UI requests the PHP backend and displays the loading icon while waiting for the response.
Behind the scenes, the Laravel framework will call the Webscrare Controller to get the scraped webpage content from Abstract Web Scraping API and display it in the front end.
Here is what the scraped content looks like for example.com
That's it !!
If you have followed the steps this far then pat yourself on your back. You have triumphantly built and tested a PHP demo app for web scraping.
As you can witness, all the heavy lifting of scraping the content was taken over by the Abstract Web Scraping API, while you focussed on building the UI and backend logic for handling user requests.
FAQs
How To Scrape Data From Websites using PHP?
As a programming language suitable for building web applications, PHP is capable of web scraping. You can use an in-built PHP library to run the scraping tasks within the PHP runtime, or leverage an external service. For large-scale scraping projects, it is recommended to use a third-party API. The Abstract Web Scraping API offers a full-fledged, secure and scalable solution for web scraping. It is easy to integrate which API within a PHP application. This API also offers a free tier for basic scraping chores.
How To Crawl A Website in PHP?
There are many ways to crawl a website. You can write the business logic in PHP using the built-in cURL library to scrape the homepage of a website and then parse all the internal links to crawl additional pages. Alternatively, you can also use an API for larger websites, which might be blocking scrape requests from the same IP address. With Abstract Web Scraping API, you can undertake large-scale website crawling tasks such that the API will spread the scraping requests across a pool of large IP addresses. Moreover, it is super easy to integrate this API within PHP using cURL or other HTTP client libraries.
Which Tool is best for Web Scraping?
There are many tools available for performing web scraping. However, if you want to have complete control over it, it is better to write your own web scraping tool. You can easily build a demo web scraping web application using PHP and Abstract API. PHP handles the web app interactions and accepts web scraping requests, while the Abstract Web Scraping API does the heavy lifting of scraping and returning the scraped content. Integrating the API within a PHP application is easy with a plethora of PHP HTTP client libraries, such as cURL. The Abstract Web Scraping API offers a free tier of 1000 requests per month to try the API.
Frequently Asked Questions
What is web scraping with PHP?
Web scraping with PHP means writing PHP code to make HTTP requests to websites, retrieve the HTML response, and extract specific data from it. PHP's server-side nature makes it a natural fit for scraping tasks within existing backend applications or Laravel projects.
How do you scrape a website using PHP?
The typical approach is to send an HTTP request to the target URL, parse the returned HTML, and pull out the data you need. Using an API like Abstract's Web Scraping API simplifies this by handling the HTTP layer for you — you create a helper class that sends the target URL to the API and processes the response in your Laravel controller.
Should I use a scraping API or a PHP library like Guzzle?
PHP libraries like Guzzle or cURL work well for simple, low-volume scraping, but they struggle at scale because websites block repeated requests from the same IP. A managed scraping API handles IP rotation and geographic restrictions for you, which is the approach covered in this guide for production-level use.
Can PHP scrape websites that load content with JavaScript?
Standard PHP HTTP requests only retrieve the initial HTML served by the server — they cannot execute JavaScript or wait for dynamically loaded content. For JavaScript-heavy sites you need a browser automation tool or a scraping API that renders pages in a headless browser before returning the HTML.
How do you integrate web scraping into a Laravel application?
The guide walks through a ten-step process: set up a Laravel project with Composer, store your API key in the environment file, create a helper class to abstract the API call, add a controller to validate the URL and invoke the helper, register routes, and build a frontend form to submit URLs. Separating concerns across helpers, controllers, and views keeps the code maintainable.
What are the best practices for web scraping in PHP?
Always validate URLs before sending them to the scraper, handle exceptions for both connection failures and invalid input, and store API credentials in environment variables rather than hard-coding them. Keeping your scraping logic in a dedicated helper class makes it easy to swap out the underlying HTTP implementation without touching your controllers.
