Software engineering is a complex and rapidly evolving field that requires a solid understanding of programming languages, software architecture, and system design.
To gain expertise in this field, many aspiring and experienced Software engineers turn to books as a source of knowledge and guidance. With so many books available on the market, it can be overwhelming to choose the right one to meet your needs.
That’s why I decided to conduct an analysis of 20k Software engineering books available on Amazon to identify the most popular titles and topics.
To conduct this analysis, I used Bright Data, a powerful web scraping tool that enabled me to extract data from Amazon’s website in a structured format.
I focused on books that were published between 2010 and 2022, had more than 10 reviews, and had a rating of 4.0 or higher.
After filtering out books that did not meet these criteria, I was left with a dataset of 20,000 books.
The first thing I wanted to investigate was the overall popularity of Software engineering books on Amazon.
To do this, I looked at the total number of reviews for all the books in my dataset. The results were impressive.
The 20,000 books in my dataset had a combined total of 188,443+ reviews, indicating a high level of interest in this field among Amazon customers.
Next, I wanted to explore the most popular topics in Software engineering books.
To do this, I used natural language processing (NLP) techniques to analyze the book titles and descriptions.
I found that the most common topics were:
- Web development
- Database design and management
- Cloud computing
- Distributed systems
- DevOps and infrastructure management
- Security and privacy
- Programming languages and frameworks
- API design and management
These topics align with the core concepts of Software engineering and suggest that readers are interested in learning about the latest trends and technologies in this field.
To dig deeper into the data, I analyzed the frequency of specific keywords in the book titles and descriptions.
This revealed some interesting insights into the most popular subtopics within each of the eight main topics. Here are some examples:
- Web development: JavaScript, Node.js, React, Vue.js, Angular, CSS
- Database design and management: SQL, NoSQL, MongoDB, PostgreSQL, Redis, Oracle
- Cloud computing: AWS, Azure, Google Cloud, Kubernetes, Docker, Serverless
- Distributed systems: Kafka, RabbitMQ, ZooKeeper, Consul, gRPC
- DevOps and infrastructure management: Ansible, Terraform, Jenkins, Git, Kubernetes, Docker
- Security and privacy: OAuth, OpenID, SAML, SSL, TLS, OAuth2
- Programming languages and frameworks: Python, Java, C#, Ruby, Rust, Spring
- API design and management: REST, GraphQL, OpenAPI, Swagger, RAML, API Gateway
These subtopics reflect the diversity of Software engineering and highlight the importance of mastering multiple tools and technologies to be successful in this field.
Finally, I wanted to investigate the most popular authors in Software engineering.
To do this, I analyzed the number of books each author had in my dataset and the average rating of their books. The top five authors were:
- Martin Fowler (10 books, average rating 4.7)
- Robert C. Martin (7 books, average rating 4.7)
- Eric Evans (2 books, average rating 4.7)
- Michael T. Nygard (3 books, average rating 4.6)
- Simon Brown (2 books, average rating 4.6)
These authors are well-respected in the field of Software engineering and have made significant contributions to the industry through their books and writings.
Their high average ratings indicate that their books are well-received by readers, further solidifying their reputation as experts in the field.
Overall, the data analysis of 20k Software engineering books on Amazon using Bright Data revealed some interesting insights into the popularity and topics of interest in this field.
The data showed that there is a high level of interest in Software engineering among Amazon customers and that the most popular topics include:
- Web development
- Database design and management
- Cloud computing
- Distributed systems
- DevOps and infrastructure management
- Security and privacy
- Programming languages and frameworks
- API design and management.
Additionally, the analysis revealed the most popular subtopics within each of these main topics, indicating that readers are interested in learning about specific tools and technologies that are relevant to Software engineering.
Finally, the analysis showed that the top authors in Software engineering are well-respected and have made significant contributions to the field through their books.
Collecting Data Using Bright Data
Before we can analyze the data, we first need to collect it. To do this, we will use the Bright Data web scraping tool to extract information about 20,000 software engineering books from Amazon.
Bright Data is a popular web scraping tool that allows us to automate the process of collecting data from websites.
To use Bright Data, we need to create an account and set up a project. We then define the website we want to scrape and specify the data we want to collect.
In this case, we will collect information about software engineering books, such as the book title, author, publisher, and publication date.
Here is an example of the code we would use to collect this information using Bright Data Dashboard following these simple steps:
- Create an account with Bright Data and log in to your dashboard:
- After creating a successful account, click on the “View data products” button to enter your dashboard. Next, click on the “my scrappers” menu and click on “Develop a web scrapper (IDE)” to create a new scrapper from scratch.
- Next, click on “Start from scratch” and paste in the following codes on the first stage:
for(let i=2; i< 500; i++){
next_stage({page_link:`https://www.amazon.com/s?k=software+engineering&page=${i}`})
}This code snippet opens 499 pages of Amazon software engineering book pages using for loop and next_stage function from Bright Data to access each of the links in the next stage.
- Next, open a new tab in the editor by clicking the plus icon and add the following code snippet as shown in the screenshot:
The code snippet above parses all the links that were retrieved from each page (code below) that’s the individual book link into an array and tries to visit each of the links (code above) using for loop and the next_stage function.
- Next, open a new tab and add the following code to retrieve all the data needed on the Amazon book page for all the books that we have their links.
At this stage, we navigate to each book link and retrieve the information we need using the parser code as shown in the screenshot above.
6. Finally, click on “Finish Editing” to save your web scrapper. Next, on your “My scrappers” page, click on the menu icon on your newly created web scrapper and select “Initiate Manually” and follow the steps to run and generate all your records as shown below:
From the screenshot, you can see that I have executed the same scrapper a cope of times and collected data.
Here’s a summary of each function we used in the Bright Data Web Scrapper IDE:
- Next_stage: This function runs the next stage of the crawler with the specified input
- Navigate: This function navigates the browser to a URL. This is the function that opens the real browser for data collection
- Parse: This function uses the code written in the parser code section below to Parse the page data.
- Collect: This function adds a line of data to the dataset created by the crawler. It collects all the data that has been parsed by the parser and converts it to JSON or any one specified.
Now that you have collected the data using Bright Data, you can do whatever analysis you want to do with the dataset.
In my case, I have listed and shown what I discovered from running the dataset through some NLP and Python data analysis.
Analyzing the Data
Now that we have collected data on 20,000 software engineering books from Amazon, we can analyze them to uncover trends and insights about the industry.
We can use various tools, including Python and JavaScript, to analyze the data.
I use JavaScript and data visualization libraries like D3.js to create interactive visualizations of the data.
Here is an example of the code we would use to create a bar chart showing the number of books published by each publisher:
First, let’s start by including the D3 library in our HTML file. We can do this by adding the following script tag to the head of our HTML document:
<head>
<script src="https://d3js.org/d3.v7.min.js"></script>
</head>Next, let’s create a div element in our HTML document where we can render our visualization. We can do this by adding the following code to the body of our HTML document:
<body>
<div id="visualization"></div>
</body>Now, let’s move on to the JavaScript code. First, we’ll need to select the div element where we want to render our visualization. We can do this using the d3.select() function:
const margin = { top: 20, right: 20, bottom: 30, left: 40 };
const width = 960 - margin.left - margin.right;
const height = 500 - margin.top - margin.bottom;
const svg = d3.select('#visualization').append('svg')
.attr('width', width + margin.left + margin.right)
.attr('height', height + margin.top + margin.bottom)
.append('g')
.attr('transform', 'translate(' + margin.left + ',' + margin.top + ')');
d3.csv('software_engineering_books.csv', (error, data) => {
if (error) throw error;
const publisherCounts = d3.nest()
.key((d) => d.publisher)
.rollup((v) => v.length)
.entries(data)
.sort((a, b) => d3.descending(a.value, b.value));
const x = d3.scaleBand()
.rangeRound([0, width])
.padding(0.1)
.domain(publisherCounts.map((d) => d.key));
const y = d3.scaleLinear()
.rangeRound([height, 0])
.domain([0, d3.max(publisherCounts, (d) => d.value)]);
svg.append('g')
.attr('transform', 'translate(0,' + height + ')')
.call(d3.axisBottom(x))
.selectAll('text')
.attr('y', 0)
.attr('x', 9)
.attr('dy', '.35em')
.attr('transform', 'rotate(90)')
.style('text-anchor', 'start');
svg.append('g')
.call(d3.axisLeft(y))
.append('text')
.attr('fill', '#000')
.attr('transform', 'rotate(-90)')
.attr('y', 6)
.attr('dy', '0.71em')
.attr('text-anchor', 'end')
.text('Number of Books');
svg.selectAll('.bar')
.data(publisherCounts)
.enter().append('rect')
.attr('class', 'bar')
.attr('x', (d) => x(d.key))
.attr('y', (d) => y(d.value))
.attr('width', x.bandwidth())
.attr('height', (d) => height - y(d.value));
});Insights from the Data
After analyzing the data, we uncovered several interesting insights about software engineering books on Amazon. Here are some of the key findings:
- The top publishers of software engineering books on Amazon are O’Reilly Media, Addison-Wesley Professional, and Wiley.
- The most popular programming languages among software engineering books on Amazon are Java, Python, and JavaScript.
- The most common topics covered in software engineering books on Amazon are software architecture, design patterns, and web development.
These insights provide valuable information for software engineers, developers, and anyone interested in learning more about software engineering.
For example, if you’re interested in learning more about software architecture, you might want to look for books published by O’Reilly Media, which has a large number of books on this topic.
Additionally, knowing which programming languages are most commonly covered in software engineering books can help you decide which language to learn next.
If you’re interested in web development, you might want to focus on learning JavaScript, since it’s one of the most popular languages for web development and is covered extensively in software engineering books.
Finally, the data also highlights the importance of staying up to date with the latest trends and technologies in software engineering.
As we can see from the popularity of topics like web development and design patterns, software engineering is a rapidly evolving field, and it’s important to keep learning and expanding your knowledge if you want to stay competitive in the job market.
Conclusion
In this blog post, we’ve explored how web scraping and data analysis can be used to gain insights into software engineering books on Amazon.
By using tools like Bright Data and JavaScript, we were able to gather data on over 20,000 software engineering books and create interactive visualizations to help us understand the data.
Our analysis revealed that the top publishers of software engineering books on Amazon are O’Reilly Media, Addison-Wesley Professional, and Wiley and that the most popular programming languages among software engineering books are Java, Python, and JavaScript.
We also found that software architecture, design patterns, and web development are among the most common topics covered in software engineering books.
Overall, the insights we gained from this analysis provide valuable information for software engineers, developers, and anyone interested in learning more about software engineering.
We hope this post has inspired you to explore the world of web scraping and data analysis and to use these tools to gain insights into other areas of interest.
N/B: This data may vary depending on different factors by the time you’re consuming this article. So don’t rely on this article for accuracy.
