The advancements in web development and its applications are not as measurable as hardware and more classical technologies. We have readily observed the commercially available maximum storage capacities of hard drives and cloud storage double every few years, however, it is widely believed that the advancements in web development over the last decade have followed Moore’s Law along the same aggressively exponential trajectory.
Back in the early 2000’s, chunky HTML and PHP were leading the scene with many applications requiring little more than the PHP 4.x standard library. Now we can build and deploy full applications in a matter of minutes with PaaS providers.
Now we can build a tool in Java to scrape financial data with little more than a decent Java web scraping tutorial. Well we have compiled that very tutorial for you. So be sure to take full advantage, because using Java can be an excellent option for finding data from your favorite financial websites!
Read on and learn the steps required to extract financial information from websites.
Use the Right Software
When using Java to scrape financial data, you should install Java 8, Maven, and a text editor. You’ll also need to decide which tool in Java to use for web scraping. JSoup is an excellent option because it’s open-source and it’s beginner-friendly.
However, you can also use Jaunt, which you can use for extracting data. It can also work with HTTP requests while supporting Java. HTMLUnit is another great framework to use when scraping financial data.
For these steps on how to scrape financial data with Java, we will use JSoup. You can use this method with Maven, and you can use it to learn about stocks and other financial reports.
Generate the Project
Once you have all of the right software ready, you can generate the web scraping project in Maven. You can use this command in Java 8 to check for Maven:
$ mvn –version
If you have Maven on your computer, the output from the command should contain the version, location, Java version, default locale, and the OS name. The specifics can depend on the computer you use and the versions of Java and Maven you have.
After you make sure you have Maven on your computer, you can use a command to generate the project. Here’s what you’ll need to use:
$ mvn archetype:generate -DgroupId=com.codetriage.scraper -DartifactId=codetriagescraper -DarchetypeArtifactId=Maven-archetype-quickstart -DinteractiveMode=false
$ cd [website]scraper
However, instead of using ”[website]scraper,” you can include the name of the website that you want to get financial data from. The website could be for the stock market or a specific company.
After you run the command, it will generate a project for your scraper. The folder it creates will have a pom.xml file that has project details and dependencies. You will use this folder to implement the dependency for JSoup.
The file will also need to have a plugin that allows Maven to add project dependencies to the jar file. And you can then use the command “java -jar” to run a jar file.
Delete Dependencies
After you set everything up, go into a pom.xml file. You will need to delete the section with the existing dependencies. Add the following code to the section to change the plugin configurations and the dependencies.
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!– our scraping library –>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.3</version>
</dependency>
</dependencies>
<build>
<plugins>
<!–
This plugin configuration will enable Maven to include the project dependencies
in the produced jar file.
It also enables us to run the jar file using `java -jar command`
–>
<plugin>
<groupId>org.apache.Maven.plugins</groupId>
<artifactId>Maven-shade-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation=”org.apache.Maven.plugins.shade.resource.ManifestResourceTransformer”>
<mainClass>com.[website].scraper.App</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
As with the initial command, you can change “[website]” to be the title of the website you’re web scraping.
After you enter the information, you can run these commands to make sure that everything works.
$ mvn package
$ java -jar target/codetriagescraper-1.0-SNAPSHOT.jar
If everything is correct, the console will show “Hello World!” which means you can start building your web scraper.
Inspect the Website
If you haven’t already, now’s the time to choose which financial data webpage you want to scrap. You will need to go to the website, and you can use whichever browser you prefer.
On both Chrome and Safari, you can right-click to get the “Inspect” option. You will then be able to see the development tools for the website you’re on.
At this point, you can look for any tags or syntax that include financial data. You can scroll through the code and see what areas of the page it highlights. Once you land on some financial data, you can search for any other examples of it.
Finding the financial data in the HTML can take a while if it’s a long page. However, you can search for numbers in the code to narrow your search.
After you find all of the data, leave up the code so that you can reference it. Then, you can move on to web scraping with Java.
Pulling Stock Prices
You can make this process quicker by hovering over the stock price on the web page itself. Then, the code will highlight the syntax for it. You can usually find the stock price in a span class.
Finding More Data
If you want to use web scraping with Java to find more detailed financial data, you can also start by highlighting part of what you want. Then, you can see if it is a div class or uses another syntax. Then, you can search for all div class instances in the code.
Build the Scraper
Head to your Java program and open the file with the name App.java. The file will show the package as well as any imports, such as JSoup and various nodes.
At the top, you should see code like:
package com.[website].scraper;
Of course, you should see the title of your website instead of “[website]” on your computer. Then, you can add the following code if you don’t see it there already:
import java.io.IOException;
Leave a line between the package and after the IOException import. This and the different JSoup classes will help you with web scraping, so you should make sure the following code shows up below the first import line:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
importorg.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class App {
public static void main(String[] args) {
System.out.println( “Hello World!” );
}
}
This will help you start to organize everything.
Modify the Main Function
Once you have your code and imports ready, you can modify the main function of your project so that it can scrape financial data. First, you should use some code to print the web page title.
public static void main(String[] args) {
try {
// Here we create a document object and use JSoup to fetch the website
Document doc = Jsoup.connect(“https://www.codetriage.com/?language=Java”).get();
// With the document fetched, we use JSoup’s title() method to fetch the title
System.out.printf(“Title: %sn”, doc.title());
// In case of any IO errors, we want the messages written to the console
} catch (IOException e) {
e.printStackTrace();
}
}
Now, you can save your file. Then, write this command to verify the existing code:
$ mvn package && java -jar target/codetriagescraper-1.0-SNAPSHOT.jar
If everything is correct, you should see something telling you the build is a success. You should also see the total time to build and when it is finished. The last line of the results will contain the website title.
If something isn’t there, double-check your code. You may need to change the website name, edit the imports, or something else.
Extract the Elements
Once you verify everything looks good, go back to your main project. Then, you can extract the elements from the syntax that you found when you inspected the website. You can use code similar to the following:
public static void main(String[] args) {
try {
// Here we create a document object and use JSoup to fetch the website
Document doc = Jsoup.connect(“[website]”).get();
// With the document fetched, we use JSoup’s title() method to fetch the title
System.out.printf(“Title: %sn”, doc.title());
// Get the list of [syntax]
// Format and print the information to the console
System.out.println(repositoryTitle + ” – ” + repositoryIssues);
System.out.println(“t” + repositoryDescription);
System.out.println(“t” + repositoryGithubLink);
System.out.println(“n”);
}
// In case of any IO errors, we want the messages written to the console
} catch (IOException e) {
e.printStackTrace();
}
}
Finally, you can run another command to view the results of web scraping with Java.
$ mvn package && java -jar target/codetriagescraper-1.0-SNAPSHOT.jar
Then, you will have the financial data that you want or need, and you can do whatever you like with it.
Web Scraping Review
Web scraping with Java is an excellent way to extract financial data regarding stocks and company balance sheets. As long as you have Java 8, Maven, and a text editor, you can use these steps on almost any website.
Keep these steps in mind when you want to use web scraping. That way, you can do so relatively quickly.