Recipe Creator Data Miner
How to write Data Miner recipes with Recipe Creator
Writing Recipes with
Recipe Creator
Recipe Creator is a feature inside Data Miner that allows you to create your own recipes for any website. In this sections
you will learn about Recipe Creator basics, selecting rows, selecting columns and Next Page Pagination. Watch the videos
below to begin.
Intro and Row Selector Video
Creating rows is an important step in scraping data. It tells Data Miner how to organize your output file. This step must
be done before creating columns. Rows are not required when creating Detail Recipes. Watch the video below to learn how to
open Recipe creator and create rows using the Easy Row Finder tool. For Detail recipes continue to Column Selector video
below.
* For tips on using the Advanced Finder please watch the video towards the bottom of this page.
Page Set-Up Video
optional
Some pages will require additional automation such as clicking a button or scrolling to the bottom of a page in order to
scrape the data.
The following video will show you how to set up some example page-set up automation.
*Requires a paid plan
Column Selector Video
Creating columns is the step where you tell Data Miner what data you want to extract. Watch the video below to learn how
to create columns using the Easy Row Finder tool.
Nav Selector Video
Nav or Next Page Pagination is what allows the recipe to click to the next page and automatically scrape the data. This
feature must be added to a recipe before it can work and Nav can only be added to List page recipes.
Advanced Finder Video
When sites have more complex HTML or the Easy Finder process is not working you may need to use our Advanced Finder tool.
This tool allows you to manually choose selectors and type in custom code to make your selectors more powerful.
Modify Columns Video
You have data that needs clean up such as removing a “mailto:” prefix or if you need to split a name use the Modify
options.
Try on your own!
Advanced Custom Selectors:
Selectors
Example
Meaning
(Space). industry strong
Separate selectors by a space to travel down elements.,
h2, p
Commas combine elements into one column. So the h2 tag data and the p tag data will be in the same column.
~
span~
Selects all the next elements as long as they all have the same parent or container.
+
span+
Selects the next one element as long as they have the same parent tag or container. :contains(” “)
div:contains(“Email”)
Recipe Creator will find the word inside quotations for any div tag on the page and select it. :first
h2:first
Recipe Creator will always find the first h2 tag on the page and select it. :last
h2:last
Recipe Creator will always find the last h2 tag on the page and select it. :eq()
h2:eq(2)
Recipe Creator will always find the second h2 tag on the page and select it. Recipe Creator can find any number
as long as there are that many on the page. (Use eq() only when numbering will be consistent between pages):has(” “)
div:has(“dress”)
Recipe Creator will find all the divs that have the p tag and the address class. Combine with:eq() to specify
which div.
[selector=”value”]
[itemdrop=”address”]
When classes or tags aren’t available but sites have alternative attributes in the HTML. Use them inside square
brackets for your selectors.
class tag. address span
When the parent container has the class, but you need data inside a child element. Select the suggested class of
the parent and then type the tag containing the data with a space in-between.
The Dynamic Data Scraping Duo: Data Miner + Recipe Creator
Stumbling upon useful information on the web (whether intentional or not) is a critical part of all our daily workflow. What you choose to do with those resources and how you later choose to leverage them directly correlates with the efficiency of your process.
Some of us add helpful sites to our Bookmarks, save webpage snippets to our Pocket or Evernote, &/or frequently exercise our copy/paste fingers. Others, well, others extract the data and create a searchable database of its intel…
When I first learned of web scraping/data mining, my internal “Hacker Alert! ” alarm went off. I quickly learned, however, that (ethically) scraping a web page’s data is far from hacking – in fact, coding knowledge isn’t even required!
Meet Data Miner… a FREE Chrome extension that helps you scrape information from ANY website into a spreadsheet. Extraction criteria is defined by a set of instructions called a recipe, which specifies where key information is contained on a site, as well as how to transform it into a desirable, structured format.
In addition to the 50, 000+ pre-made, public recipes that Data Miner’s 90, 000+ users have made available for 1-click scraping on 15, 000+ websites, Data Miner also offers CUSTOM recipe creation using their FREE Chrome extension, Recipe Creator.
Recipe Creator‘s user-friendly interface guides you through the recipe creation process via Information snippets and accessible video tutorials.
Clicking the Help icon reroutes you to Data Miner’s Live Support, whose offerings include talking with an expert, screen sharing, participating in group learning, walking through live demos, & even creating (paid) recipes.
Data Miner also provides a learning resource, Getting Started with Data Miner, that features step-by-step “How to” instructions for Scraping Web Pages and Basic Tables, Multi Page Scraping and Automation, & Writing Custom Recipes. Video tutorials accompany these lessons, as well (all of which are housed on the Data Miner YouTube Channel).
Tips for Creating Custom Recipes – Choosing a Unique Identifier for a Selected Element
Begin by checking the fewest element attribute boxes possible.
Don’t be afraid to play around with the “Select Parent” functionality.
Access your browser’s Developer Tools to inspect the selected element and cross-reference Data Miner’s Recipe Creator Selector Tricks, Tips, & Scenarios.
Examples:
Other Tips – Utilize Data Miner’s Community!
Bounce questions and ideas off the Data Miner User Community’s Facebook Group.
Expand your knowledge by following @DataMinerIO on Twitter.
Actually reach out to Data Miner’s Support Team – They have a warm, customer-centric attitude and are more than willing and able to help you!
Visit Data Miner’s website to explore additional features and resources.
Happy Scraping, y’all!
Data Miner FAQ
Data Miner FAQ:
By using Data Miner you explicitly agree to the terms of use of Data Miner website
and to the subscription plan terms. Data Miner subscription plans start immediately
when you sign up for a plan and continue monthly until you cancel your plan from the
account page. Please read our terms of use and feel free to ask us any
question you may have before using our tool.
Can I share or use public recipes for LinkedIn?
Some sites, like LinkedIn, have terms of service that disallow scraping. When we have been asked by LinkedIn,
and we complied by not allowing sharing of recipes on our site and by not encouraging or coaching our customers in
violations of that site’s term of use in any way.
Does Data Miner work on any page?
Data Miner can work on any HTML page on any site. If the page is a pdf or Word doc or zip file, Data Miner
will not work.
Can Data Miner get email and phone number from any web site?
Data Miner tool simply transforms the HTML content of a page into a CSV format. If the page you are scraping does not have some information,
like email or phone number, we cannot get that for you. Data Miner tool can not extract any info that is not
on the page that you cannot see yourself.
Is my data safe?
Data Miner does works only on your computer. Besides page credits counting information, we do not send any
data to our servers. Some companies use your access to various sites to make their own database of contacts
and other information and then sell that to other users. We see this as a huge violation of privacy and an
inappropriate if not illegal behavior.
Are Extensions Safe?
Extensions are very safe because they can’t run code on your machine and they can be uninstalled completely
from Chrome. Google has a lot of vested interest to keep the browser ecosystem clean and safe so you can
rely on them. Extensions are much safer than installing software on your machine, which can do anything
and you can’t uninstall it completely.
What are Recipes?
Recipes contain name and position of HTML elements on a web page and tell Data Miner what data to extract.
Recipes do not contain actual data or private information about you. When you visit a website, Data Miner
automatically filters thousands of recipes and shows only the ones that are appropriate for the site
you are currently viewing.
What are Private Recipes?
Private Recipes are only visible to you. In contrast to Public recipes which are recipes written and shared by other
users and are visible to everyone. By default the recipes you create and save are private. If you have a paid subscription
plan you can make your own recipes public so that others can see them.
What’s Custom Javascript for?
Data Miner can execute a Javascript function that you provide after it has scraped your data.
This adds a great degree of flexibility to what you can with Data Miner. For example, you can clean
up the text you extracted, split names into first and last name column, or click on “see more”
buttons on the page and much more.
What are Automated Scrape Jobs?
Say you have 100 URLs representing the pages of a product catalog. You can use Data Miner to automatically
open each URL and scrape information from it. All the data from different pages will be aggregated into
single data collection which you can save when the automated process is finished. Remember no data is stored
in Data Miner servers. So you will need to save your data after you are done with the automated scrape job.
To learn more about Automate scrape jobs. See an example of how you can use a Job to
scrape a list of professionals on Data Miner’s practice page.
Plan Terms
Can I change my plan at any time?
Of course. You can upgrade and downgrade at any time. We do not lock you in long term plans. You can downgrade
to the free plan at any time you like. We offer a free plan to everyone so that you can evaluate Data Miner
and make sure it works for your specific scenario before upgrading to a paid plan.
What if I decide to cancel?
You may cancel at any time. Simply downgrade to the free plan and you will not be bill the next monthly billing period.
Please note that we
can not issue you a refund for a partial month. Once you cancel your plan, you will not be billed any longer and you will lose any
unused credits left in your account. So it is to your interest to cancel your plan at the end of the month once you have
exhausted all your scraping credits.
Please feel free to contact us for help regarding any billing issues.
When you downgrade to the free plan any private recipes you had may be marked as public.
Do my unused scraping credits carry forward?
They do not. Every month you can scrape up to the maximum scraping limit of your account. At the start of every billing period
your scrape counter is set to zero.
Do you offer support?
We offer daily interactive live support session in our free Office Hours to all our users in a group
live meeting session.
In addition we offer phone support to our Enterprise plan customers.
If you are subscribed to that plan, you can schedule a
phone call with us for one on one support during regular business hours in US Pacific Time Zone (UTC-08:00).
We support our Business plan customers with email and we have extensive self help documentation our Free and
Solo plan customers.
I can’t get Data Miner to work for me at all?
We offer a 500 page scrapes per month for free so that you can become familiar with Data Miner service. However
if you have major issues after you subscribe please contact us and we will consider issuing you a refund if
you have not been able to use Data Miner at all. We will consider your request on case on a case by case basis.
Do I get 500 free page scrapes per month in addition to my paid subscription plan?
No. The first time you upgrade from free plan to the paid plan we erase the number of page scrapes you have
done under the free plan and you start from zero page scrapes under your paid plan.
I have more questions. Whom do I ask?
Try searching for what you need in our Help page. If you do not find what you’re looking for,
you can email us at for more help.
Secure Payments:
All our credit card processing is securely handled using an encrypted HTTPS connection by
