Web Scraping with Webiny, Puppeteer, React, and SyncFusion

With that done, we would build the frontend application to consume the scraped data.

Setting Up Frontend

The frontend application would live in the frontend sub-directory.

Prerequisites

• Node ≥ v16.14
• SyncFusion
• Tailwind CSS
• Apollo Client

Use the latest version of Node for best results.

nvm install 20

In the frontend directory, create a new React app, using this command:

npx create-react-app job_board

Next, in job_board, we can install the other project dependencies. Run the command below to install the SyncFusion dependencies needed for the project. SyncFusion is a React UI component library that helps in speeding up code development.

npm install @syncfusion/ej2-base @syncfusion/ej2-buttons 

You need to create a SyncFusion account and obtain a license key to use SyncFusion. You can create a Free Community Account. It might take up to a week to get a license key. However, SyncFusion would provide you with a temporary license key to use in the meantime.

If your React app doesn’t include out-of-the-box, Tailwind configuration files (tailwind.config.js and postcss.config.js). Then, you want to install Tailwind and make the configuration.

To install, run this command:

npm install tailwindcss postcss autoprefixer

To configure, run this command in the project root directory (job_board):

npx tailwindcss init -p

Next, create a postcss.config.js in the root directory and add this in.

module.exports = {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
};

Next, ensure Tailwind is imported into your index.css in the root directory.

/* styles/globals.css */
@import 'tailwindcss/base';
@import 'tailwindcss/components';
@import 'tailwindcss/utilities';

/* Your other styles go here */

Next, ensure tailwind.config.css is configured properly to include the necessary paths.

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
    './src/pages/**/*.{js,ts,jsx,tsx,mdx}',
    './src/components/**/*.{js,ts,jsx,tsx,mdx}',
    './src/app/**/*.{js,ts,jsx,tsx,mdx}',
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

Next, we would install Apollo Client. This would be used for handling API requests.

npm install @apollo/client graphql

This is how your finished project directory should look. Throughout this tutorial, our focus will be mostly on the src directory. So go ahead and create the api , components, and pages directories, as well as their corresponding JSX files.

frontend
└── job_board
    ├── README.md
    ├── .env.local
    ├── node_modules
    ├── package-lock.json
    ├── package.json
    ├── postcss.config.js
    ├── public
    ├── src
    │   ├── App.css
    │   ├── App.js
    │   ├── App.test.js
    │   ├── api
    │   │   ├── client.js
    │   │   └── scraper.js
    │   ├── components
    │   │   ├── HeroSection.jsx
    │   │   ├── JobCard.jsx
    │   │   ├── Search.jsx
    │   │   └── index.jsx
    │   ├── index.css
    │   ├── index.js
    │   ├── pages
    │   │   ├── JobList.jsx
    │   │   └── index.jsx
    │   ├── reportWebVitals.js
    │   └── setupTests.js
    └── tailwind.config.js

Integrating Webiny CMS for fetching data using GraphQL and Apollo Client

In the api directory, we would configure the Job Board App for sending get requests to Webiny. In client.js , we would set up the API configuration. While scraper.js will hold the GraphQL query.

Update client.js with the code below. In this case, the API endpoint that would be used is the Webiny Read API copied from API playground in the Webiny Admin.

//client.js

import { ApolloClient, InMemoryCache, createHttpLink } from '@apollo/client';
import { setContext } from '@apollo/client/link/context';

const httpLink = createHttpLink({
    uri: process.env.REACT_APP_GRAPHQL_ENDPOINT, //endpoint goes in here
});

const authLink = setContext((_, { headers }) => {
    const token = process.env.REACT_APP_GRAPHQL_TOKEN; //token goes in here

    return {
        headers: {
            ...headers,
            authorization: token ? `Bearer ${token}` : '',
        },
    };
});

const client = new ApolloClient({
    link: authLink.concat(httpLink),
    cache: new InMemoryCache(),
});

export default client;

Add the Read API URL and Token to .env.local in the root directory.

REACT_APP_GRAPHQL_ENDPOINT = WEBINY_READ_API //naming of env variables must start with REACT_APP!
REACT_APP_GRAPHQL_TOKEN = WEBINY_TOKEN

Next, update scraper.js with the code below:

// LIST_WEB_SCRAPERS_QUERY.js
import { gql } from '@apollo/client';

export const LIST_WEB_SCRAPERS_QUERY = gql`
  query webScraper {
    listWebScrapers (limit: 115) {
      data {
        title
        image
        location
        salary
        company
        datePosted
        jobLink
      }
    }
  }
`;

Next, we would modify index.js in the src directory to enable the API configuration we just created.

//index.js

import React from 'react';
import ReactDOM from 'react-dom/client';   //import this!
import './index.css';
import App from './App';
import { ApolloProvider } from '@apollo/client'; //import this!
import client from './api/client';                 // import this!
import reportWebVitals from './reportWebVitals';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <ApolloProvider client={client}> {/*wrapping react with Apollo*/}
    <React.StrictMode>
      <App />
    </React.StrictMode>
  </ApolloProvider>

);
reportWebVitals();

With that done, the app is now ready to send API requests. We would make one last change to index.js. We would add in our SyncFusion license key. Update .env.local in the root directory by adding your license key.

REACT_APP_LICENSE_KEY = LICENSE-KEY  //naming of env variables must start with REACT_APP

Then, modify index.js to include this:

//index.js

....
import { registerLicense } from '@syncfusion/ej2-base';

// Register the license key
registerLicense(process.env.REACT_APP_LICENSE_KEY);

....

Creating Routes for the Application

To create routes for our App, we would update the App.js. For this app, we would define only one route. This would be the Job board page for displaying, searching, and filtering jobs.

import React from 'react';
import { BrowserRouter as Router, Routes, Route } from 'react-router-dom';
import { JobList } from './pages'; //we would implement this!
import { HeroSection } from './components'; //we would implement this!

// Import Syncfusion CSS
import '@syncfusion/ej2-base/styles/material.css';
import '@syncfusion/ej2-buttons/styles/material.css';

const App = () => {
  return (
    <Router>
			<HeroSection />   {/*we would implement this!*/}
      <Routes>
        <Route path="/" element={<JobList />} /> {/*this is the route*/}
      </Routes>
    </Router>
  );
};

export default App;

Creating Components and Pages

Create a Job List Page

The Job List page displays a list of jobs that can be searched and filtered using job title, job location, company, and date posted. In pages, create JobList.jsx. It would hold the logic for the Job List page.

// JobList.jsx
import React, { useEffect, useState } from 'react';
import { JobCard, Search } from '../components'; //we would implement this!
import { useQuery } from '@apollo/client';
import { LIST_WEB_SCRAPERS_QUERY } from '../api/scraper'; //import this!

const JobList = () => {
    const [jobs, setJobs] = useState([]);
    const [filteredJobs, setFilteredJobs] = useState([]);
    const [uniqueCompanies, setUniqueCompanies] = useState([]);
    const { loading, error, data } = useQuery(LIST_WEB_SCRAPERS_QUERY);

    useEffect(() => {
        if (data) {
            setJobs(data.listWebScrapers?.data || []);
            setFilteredJobs(data.listWebScrapers?.data || []);

            // Extract unique company names
            const companies = [...new Set(data.listWebScrapers?.data?.map((job) => job.company))];
            setUniqueCompanies(companies);
        }
    }, [data]);

    if (loading) {
        return <p>Loading...</p>;
    }

    if (error) {
        return <p>Error: {error.message}</p>;
    }

    const handleSearch = ({ searchTerm, locationFilter, companyFilter, datePostedFilter }) => {
        const filtered = jobs.filter((job) => {
            const titleMatch = searchTerm ? job.title.toLowerCase().includes(searchTerm.toLowerCase()) : true;

            if (titleMatch) {
                const locationMatch = !locationFilter || job.location.toLowerCase().includes(locationFilter.toLowerCase());
                const companyMatch = !companyFilter || job.company.toLowerCase().includes(companyFilter.toLowerCase());

                if (datePostedFilter === 'anytime') {
                    return locationMatch && companyMatch;
                }

                if (job.datePosted.includes('h')) {
                    // If 'h' is present, classify as "Last Day"
                    return locationMatch && companyMatch && datePostedFilter === 'lastDay';
                }

                if (job.datePosted.includes('d')) {
                    // If 'd' is present, convert to number
                    const numericValue = parseInt(job.datePosted);

                    if (numericValue < 1) {
                        return false; // Exclude jobs with non-positive numeric values
                    } else if (numericValue < 7) {
                        // Classify as "Last Week"
                        return locationMatch && companyMatch && datePostedFilter === 'lastWeek';
                    } else if (numericValue < 14) {
                        // Classify as "Last Two Weeks"
                        return locationMatch && companyMatch && datePostedFilter === 'lastTwoWeeks';
                    } else if (numericValue < 30) {
                        // Classify as "Last Month"
                        return locationMatch && companyMatch && datePostedFilter === 'lastMonth';
                    } else {
                        // Classify as "Last Two Months"
                        return locationMatch && companyMatch && datePostedFilter === 'lastTwoMonths';
                    }
                }

                // If no 'h' or 'd', default to numeric value
                return locationMatch && companyMatch;

            }

            return false;
        });

        setFilteredJobs(filtered || []);
    };

    return (
        <div className="container mx-auto px-16 py-8">
            <Search onSearch={handleSearch} uniqueCompanies={uniqueCompanies} />
            <div className="flex flex-wrap justify-start -mx-4 items-stretch">
                {filteredJobs.length > 0 ? (
                    filteredJobs.map((job, index) => (
                        <div key={index} className="w-1/4 mb-8 px-2">
                            <JobCard job={job} className="flex-1 h-full" />
                        </div>
                    ))
                ) : (
                    <p>No matching jobs found.</p>
                )}
            </div>
            <div className="flex justify-center mt-4">
                {/* Pagination component */}
            </div>

        </div>
    );
};

export default JobList;

Create an index.jsx in the pages directory. Add this:

export { default as JobList } from './JobList';

Create a Job Card Component

The Job Card component would contain the job data. It would be used to populate the Job List page. In the components directory, create JobCard.jsx, and modify it as given in the code below:

//JobCard.jsx
import React from 'react';
import { ButtonComponent } from '@syncfusion/ej2-react-buttons';

const JobCard = ({ job }) => {
    return (
        <div className="max-w-md mx-auto mt-4 shadow-md rounded-md p-6 bg-white flex flex-col justify-between h-full">
            <div>
                <img src={job.image} alt={`${job.company} Logo`} className="mb-4 h-16 w-auto mx-auto" />
                <p className="text-xl font-semibold mb-2">{job.title}</p>
                <p className="text-gray-600 mb-2">{job.company}</p>
                <p className="text-gray-600 mb-2">{job.location}</p>
                <p className="text-gray-600 mb-2">{job.datePosted}</p>
                <p className="text-gray-600 mb-2">{job.salary}</p>
            </div>
            <ButtonComponent
                onClick={() => window.open(job.jobLink, '_blank')}
                target="_blank"
                rel="noopener noreferrer"
                isPrimary={true} 
                className="text-blue-500 hover:underline"
            >
                View Job
            </ButtonComponent>
        </div>
    );
};

export default JobCard;

Create Search Component

The search component contains the logic for filtering job data. Create Search.jsx in the components directory.

//Search.jsx
import React, { useState, useEffect } from 'react';
import { ButtonComponent } from '@syncfusion/ej2-react-buttons';

const Search = ({ onSearch, uniqueCompanies }) => {
    const [searchTerm, setSearchTerm] = useState('');
    const [locationFilter, setLocationFilter] = useState('');
    const [companyFilter, setCompanyFilter] = useState('');
    const [datePostedFilter, setDatePostedFilter] = useState('');

    useEffect(() => {
        // Clear company filter when uniqueCompanies changes
        setCompanyFilter('');
    }, [uniqueCompanies]);

    const handleSearch = () => {
        onSearch({
            searchTerm: searchTerm.trim(),
            locationFilter: locationFilter.trim(),
            companyFilter: companyFilter.trim(),
            datePostedFilter: datePostedFilter.trim(),
        });
    };

    return (
        <div className="flex justify-between items-center mb-4 space-x-4 p-4">
            <div className="flex space-x-4">
                <input
                    type="text"
                    placeholder="Search by title"
                    value={searchTerm}
                    onChange={(e) => setSearchTerm(e.target.value)}
                    className="border p-2"
                />
                <input
                    type="text"
                    placeholder="Filter by location"
                    value={locationFilter}
                    onChange={(e) => setLocationFilter(e.target.value)}
                    className="border p-2"
                />
                <div className="w-40">
                    <select
                        value={companyFilter}
                        onChange={(e) => setCompanyFilter(e.target.value)}
                        className="border p-2 rounded-md"
                        style={{ maxWidth: "10rem" }}
                    >
                        <option value="">Select company</option>
                        {uniqueCompanies.map((company, index) => (
                            <option key={index} value={company}>{company}</option>
                        ))}
                    </select>
                </div>
                <div className="w-40 ">
                    <select
                        value={datePostedFilter}
                        onChange={(e) => setDatePostedFilter(e.target.value)}
                        className="border p-2"
                    >
                        <option value="" disabled>Select Date Posted</option>
                        <option value="anytime">Anytime</option>
                        <option value="lastDay">Last Day</option>
                        <option value="lastWeek">Last Week</option>
                        <option value="lastTwoWeeks">Last Two Weeks</option>
                        <option value="lastMonth">Last Month</option>
                        <option value="lastTwoMonths">Last Two Months</option>
                    </select>
                </div>
            </div>
            <ButtonComponent
                onClick={handleSearch}
                cssClass="bg-blue-500 text-white p-2 rounded-md hover:bg-blue-600 focus:outline-none focus:ring focus:border-blue-400 transition"
            >
                Search
            </ButtonComponent>
        </div>
    );
};

export default Search;

Create HeroSection Component

We would have a simple header for the App. Create HeroSection.jsx in the components directory.

//HeroSection.jsx
import React from 'react';

const HeroSection = () => {
    return (
        <div className="bg-blue-500 text-white p-4">
            <div className="container mx-auto">
                <h1 className="text-2xl font-semibold">Web Scraping with Webiny</h1>
            </div>
        </div>
    );
};

export default HeroSection;

With that all done, create index.jsx in the components folder. Add this:

export { default as JobCard } from './JobCard';
export { default as Search } from './Search';
export { default as HeroSection } from './HeroSection';

With that done, you can now run your Job Board App.

npm start

Conclusion

We have successfully built a Web Scraping with Webiny, Puppeteer, React, and SyncFusion. In this article we learn step-by-step how to put Puppeteer, Webiny Headless CMS, and React together for web scraping and building applications with the extracted data.

Full source code: https://github.com/webiny/write-with-webiny/tree/main/tutorials/webscraper-webiny