In the second installment of our three-part series explaining how to tell data-driven stories using import.io, this screencast shows how to extract data from a site which requires a login to access it.
The first installment demonstrated how to extract data about the richest people in America using the import.io extractor.
To follow the steps in this screencast, you will need to create a free account on M-Farm, which tracks food market prices around Kenya.
The video above explains how to create an authenticated API to M-Farm using import.io, or alternatively you can follow the steps below.
1. If you do not already have the import.io browser installed on your computer, you can download it for free from the website.
Open the browser, navigate to M-Farm’s website and click the pink 'io' button in the top right corner. Then, click 'Let's get cracking' and select the 'extractor' button on the right.
The next screen will ask you if you need to login to see the data. Click 'yes', then click on 'I'm there'.
2. Now you need to record your login information. To do this, simply click the red record button and enter your email address and password into the login form for M-Farm, then press 'sign in'.
Once you have signed in, press the stop button to stop recording. To start the extraction process press 'take me to the next step'.
3. Before you can extract the data, you first need to navigate to it on the page. Once you have done that, press 'I'm there' and then 'detect optimal settings'. The data should still be visible so click 'yes'.
Because the data on this page is laid out in a table, you can use import.io's new beta feature auto table extract.
First, click the green 'ATE' button on the left.
Then, simply hover the mouse over the table until it turns green and click it. This should highlight all of the information in the table, and you can press 'extract table'. All of the data (including the column headings) should be pulled into the table automatically.
All you need to do now is double-check that it is correct and that you are happy with the column headings. When you are satisfied, press 'I've got what I need'. Then, upload it to import.io and their tool will create an API for that data source.
4. Because you had to login to see the data on the M-Farm website, you will also need to login to see it on your dataset page.
To do this, click the pink 'query' button underneath your source and enter your M-Farm details again. The box should turn green and you should see all the data pulled back live from the site.
Free daily newsletter
- What does GDPR mean for journalists?
- Tip: Advice for working with collaborative data journalism projects
- Tip: Bookmark these tools for collaborating on investigative projects
- Tip: Take note of this advice for collaborating on data journalism projects
- Tip: Check out these data and interactive journalism resources from NICAR 2018