Dapper Help Topics

General Questions

Using Dapps in Feeds and Widgets

Creating Feeds from Websites — The Dapp Factory

Website Owners

Developers

Reading Content from Sites that Require Log-In

Troubleshooting

General Questions

What is Dapper?

Dapper is a free, web-based service that allows you to extract and use information from any website on the Internet. It allows you to create a "feed" (which we call a Dapp) for any site without programming. You can then use these feeds in a variety of ways: as an RSS feed, a widget, in Facebook applications, and more (and if you're a programmer, you can incorporate them into your applications and websites).

What is a Dapp?

Dapper works exactly as your browser does &mdash it reads content from a website's page. The difference is that instead of showing the entire page, Dapper streams a feed of just the content you select.

A Dapp is the definition of the content you want from a certain page type.

For example, let's say you create a Dapp that reads just the movie titles and thumbnails from a video website search results page, and streams them as an RSS feed. When Dapper goes to that website, it will use the Dapp to know which content to read and send in the feed.

You can change the page Dapper reads by supplying the Dapp with a different URL to go over. This new page should be the same page type and have the same structure and layout as the first page. For those knowledgeable in programming, it is possible to automate the process of calling the Dapp the for several different pages, one at a time.

How much does it cost?

The Dapper service is free and open to all. Free use is subject to several restrictions (including number of hits permitted per second, limited customer support, etc.).

I'm not a programmer, can I use Dapper?

Yes. While being a programmer allows you to build more sophisticated applications, you can still use Dapper to manipulate data from the web without knowing how to program. For example, you can easily create an RSS feed for any website, build a Google Maps mashup in a few minutes, or get alerts about changes to your favorite website.

Can I control who views my Dapp and its content?

Yes. If you mark your Dapp as private, no one else will be able to see it. If it is public, however, it is visible to the world, and any sensitive information that it contains is as well (besides usernames and passwords). Please take this into consideration when building or using Dapps. Your account information will always be private and we will not sell or provide your personal information to anyone.

Why should I sign up for an account?

Signing up for an account allows you to create Dapps. Creating an account is free and your information is private.

Can I run Dapper on my own server?

Not currently. If this is something you'd be interested in, please contact us, though we are currently not offering a standalone solution.

What about copyrights?

Dapper users have the sole obligation to respect the copyrights of content they use from the web. Dapper will not be held liable for any sort of copyright infringement, and will immediately comply with any verified request by the lawful owner of the content to cease using his content. Please see our terms of use for more information.

Using Dapps in Feeds and Widgets

How do I use Dapps?

To use a Dapp, go to a Dapp's page and choose the feed format from the dropdown. The Dapp's content will be transformed to the format you choose.

For example, if you want an RSS feed for your local newspaper site, create a Dapp for it, choose "RSS" in the Dapp's Page, and use the link we provide to subscribe to it.

Here are some more examples of how you can use Dapps:

Subscribe in your reader: RSS Feed, Filtered RSS feed, Atom Feed
Embed in your site: Flash Widget
Put on your start page: Google Gadget, Netvibes Module, FageFlake
Put on a map: Google Map
Put in a calendar: iCal
Go build something: XML, JSON, XSL, YAML, HTML, EMAIL

How often do Dapps update with new content?

Dapps are updated whenever they are requested (for the technically literate, they are "proxies"). If you request the content from the Dapp, Dapper will fetch the most recent content. Dapper does not make requests for the content when no one accesses the Dapp.

For example, if you use a Dapp as an RSS feed, it will be updated whenever your RSS reader requests the RSS feed from Dapper.

Dapper employs a ten-minute cache.

The site changed but my feed does not reflect the changes

Dapp results are cached for 10 minutes at a time, so if your Dapp is very popular and is viewed a lot, Dapp results can be up to 10 minutes old.

What is Dapp linking?

Dapper allows you to link two or more Dapps together. For example, you can provide the output of one Dapp as the input to another, subsequently building a new, complex Dapp. This process requires no programming. A simple use for Dapp linking would be to link a news site's main page and the inner article pages , and combine them into one big page. Another example for a possible use would be to take the results of an MSN search and link them into a translation Dapp in order to translate them to another language. You can access the Dapp linking feature through your Dapp's page, under "Link Dapp's output to another Dapp".

The Dapp Factory

How do I create a Dapp?

To create a Dapp, click on 'create a new Dapp' on the Dapper homepage. You'll then be taken to a wizard that includes 5 steps (See a step-by-step screencast):

  1. Start: Enter the URL for the site you want to extract content from.
  2. Collect Sample Pages: You will now be in a "virtual browser" where you supply samples of relevant pages to your samples basket. You should populate your basket with pages that look the same (i.e., have the same structure and layout), but have different content (e.g., search results pages, product pages). If there is only one unique page on the site (e.g., a homepage), you can add just that page.
  3. Select Content: Click on parts of the page to mark or unmark them, until only the content you're interested in is marked, and then save it as a field. Different content types should be saved as different fields (e.g., create one field for "Movie Title" and one field for "Movie Thumbnail").
  4. Preview Feed: See all the content you chose, and group related fields. Grouping related content fields makes sure that related content will be displayed together, like it is in the page you create the Dapp for (e.g., "Movie Title" together with "Movie Thumbnail").
  5. Save
After the Dapp is saved, you will see the new Dapp page where you can select the format of the feed content.

Does Dapper work with all web pages?

Dapper works with all web pages, excluding ones built in Flash. When working with a Flash-based website, the flash sections will be disabled, but you'll be able to work with all other sections of the website.

How does Dapper determine the default page for a Dapp to extract content from?

Dapper extracts content from one page at a time. The page that Dapper extracts content from can be changed by supplying the Dapp with a different URL (which should be similar in page structure to the default one). For those knowledgeable in programming, it is possible to automate the process of calling the Dapp the for several different pages, one at a time.

Why does Dapper ask for more than one URL in step 2 of the Dapp Factory?

The more example pages you supply, the better Dapper works. This is because have more samples allows Dapper to determine which elements recur on every page and which data is dynamic. Typically, only dynamic data is of interest, so being able to isolate this data is useful. Good sample pages are pages that have the same structure and layout, but have different content (e.g., search results pages, product pages). If the page you want content from is unique in its layout (e.g., a homepage), you can add just that page. Watch a demo that explains step 2.

Does the Dapp remember what I write inside forms?

Yes it does. By default, what you write in form inputs is hardcoded into your Dapp. In step 2 of the Dapp Factory, fill the form as you would regularly, submit it, and add the resulting page to your basket. Every time you'll view the feed, Dapper will write exactly what you wrote when you created the Dapp.

For example, let's say you create a Dapp for Google search. You search Google for "oranges", add the search results page to the basket and extract the results. Every time you'll view the feed, Dapper will run a Google search for "oranges".

If you want more freedom and customization in your final Dapp, and you do not want the input hardcoded, you should mark form inputs as input variables (you'll see a checkbox when you hover above an input field with your mouse). You'll then be able to change your choice in the final Dapp, and the Dapp output will reflect the change.

What are variable inputs?

Dapper can remeber and automatically enter content into web forms, and then bring back the content that results from the form submission. When you mark a form input as variable when creating a Dapp (see illustration below), Dapper remembers that form and later on lets you use it remotely.

Assigning variable inputs to a Dapp lets you use a website's functionality as well as its content.

For example, let's say you want to create a widget that searches google remotely. When creating the Dapp, check "Use as variable input" under the Google search box. This will tell Dapper to remember this search box and emulate it later in the widget.

Then add some sample pages with different searches to the sample pages basket, continue to select the search results titles, and create the widget in the new Dapp's page.

Every time you'll use this widget, Dapper will enter the search in Google and bring back the results. You can use these variables in RSS, Google gadgets, XML all of the other feed formats.

How do I set input variables for a Dapp?

There are two ways to set input variables:

  1. If you start your Dapp creation with a page that contains a form (e.g. Google search page), you will see a green message with a checkbox under any available input field (e.g. Google's search box) when you hover over. Check the relevant checkboxes, give the variable a name (e.g. "Search") and save it. Continue creating the Dapp and save it. You see the form inputs you've selected in the Dapp's page, and you'll be able to change the input there.
  2. Alternatively, you can define GET variables using the "input variables" feature in the "Save a Dapp" stage. You can change any part of the URL provided with a variable name, surrounded by curly brackets to let Dapper know that this is a variable.
    For example:
    If given the following URL:

    http://www.google.com/search?hl=en&q=oranges

    you can alter it to:

    http://www.google.com/search?hl=en&q={Search}

    to insert a variable called "Search".
    Continue creating the Dapp and save it. You'll see the form inputs you've selected in the Dapp's page, and you'll be able to change the input there.

What are groups? (step 4)

A group is a collection of related content (as defined by fields). Grouping fields makes sure that related content will be displayed toghether, like it was in the page you created the Dapp for.

Website Owners

I'm a website owner, why should I use Dapper?

Dapper can help you get more exposure for your website by allowing you to distribute your content in new forms and allowing others to easily consume your site and services. With Dapper, you can now easily cater to your users in new ways by allowing them to consume your content such as RSS, APIs, and all kinds of widgets.

Can I prevent my content from being used by Dapper?

While we encourage you to consider the benefits of allowing Dapper's users to access your content using our service, we certainly respect your rights as the creator of your content. How to block Dapper using robots.txt.

Developers

Is there a Dapper SDK for developers?

Yes! You can check out the available SDKs and documentation.

Can I build my own format transformer?

Yes. Format transformers are tools which transform the raw XML produced by Dapper to various useful formats. Dapper's format transformers are released as open source code and you are encouraged to use them as a basis for new and improved ones. We'll be glad to share new format transformers with Dapper's user base. Go create a tranformer: Transformers Documentation.

Extracting Content from Sites that Require Log-In

How do I create a Dapp that logs into a website?

Creating a Dapp which logs into a site is just like creating any other Dapp. Use the Dapper Browser from the "Create a Dapp" section on the main page to surf to the site you want to Dapp. Once there, navigate to the site's login form. Go through the login process in the Dapper Browser as you would on the site itself, and then add pages to your basket as usual. When you proceed to define your fields and groups, you'll be asked for your username and password. Supply the credentials you ordinarily give to the site, and Dapper will log in to the site as you, and perform its analysis.

Your username and password are not saved in the Dapp (for your security), and as such you will have to supply them whenever you use the Dapp.

How secure is Dapper with regard to supplying my username and password to remote websites?

Dapper is as secure as the remote website is, with one temproary caveat. When you submit your username and password to Dapper, they are transmitted over HTTP (not HTTPS), and as such are not encrypted. We plan to introduce HTTPS support soon. Regardless, if the remote server supports HTTPS, your credentials are sent from Dapper to the remote website using HTTPS.

I am using an existing Dapp which logs into a website. How do I send my username and password?

The best way to start is by visiting the Dapp's page. This will allow you to set up the URL to access your Dapp. Essentially you need to supply two variables when calling the Dapp, either via POST or GET. The variables are username and password. If you prefer not to send usernames and passwords around in URLs, then use POST, by using the Dapp's page to encrypt the username and password, or use our credential encryption service detailed below.

Do the Dapper SDKs support Dapps that log-in to websites?

The PHP5 SDK fully supports Dapps that log-in to websites.

Do Dapps that log-in to websites work in every Dapper feature?

Most, but not all. Currently you cannot link together Dapps that log-in to websites, and you cannot use AggregatorAid with Dapps that require a username and password.

I want to build a service that makes use of a login Dapp without resending my users' login information in the clear. How do I do that?

Dapper provides an encryption service to eliminate the need to re-send login information in the clear repeatedly. Using this service, you can encrypt each user's information once, store the encrypted values (either on your server or in a cookie on the client side) and provide the encrypted values to Dapper whenever you access the Dapp for that user. As explained above, Dapper will submit your users' credentials to the site with the highest security level that site supports. To access the credentials encryption service, call: http://www.dapper.net/encrypt- credentials.jsp with any variables and values you wish to encrypt. It is recommended to call this service via POST rather than GET (as demonstrated below) because then the credentials will not be visible in the URL. The service will respond with a JSON message that contains the variables and their corresponding encrypted values. For example, the following request:

http://www.dapper.net/encrypt-credentials.jsp?username=plato&password=Phaedrus

will return the following response in JSON:

{"password":"247754DD0F86014FF7F88AD663D1B952","username":"346BF5A295AB0806"}

which can then be stored for future use. To use these encrypted credentials, just use them as values to the respective Dapp variables inside the URL, and add "&encryptionAlg=1" to the URL, for example:



In the near future, when we introduce HTTPS calls to Dapper, the need to send login and password in the clear even once will be eliminated (if using POST).

Troubleshooting

How can I extract content from several results pages into one feed?

The extraction of content from several result pages, or "pagination", is currently not supported by Dapper. As this is a widely requested feature, we are considering adding it in the future.

Here's what you can do in the meantime:

  1. Set a separate input for each result page:
    1. Go to your Dapp's page.
    2. Inside the Dapp's page, choose a format for the Dapp, like RSS, and click go.
    3. In the menu that opens up, click the little arrow next to 'Set Input' and in the text field that opens up, write down the URL that you'd like to apply the Dapp's "rules" on.

      For example: if the default URL was

      http://www.youtube.com/results?search_query=rain&page=1

      change it to

      http://www.youtube.com/results?search_query=rain&page=2

    4. The link that Dapper generates below will now reflect output based on the same set of rules implemented on the second page. Copy that link's location, and repeat the process for the next result page.
  2. Advanced: Use a tool like Microsoft Popfly or your programming language of choice to create a loop, that will use your Dapp to collect the content from all the consecutive pages.

How do I change the RSS item title to use another link?

If you are trying to create an RSS feed from a page that doesn't link through its title, but through a different element (a "Read More" link, for example), you can change the title link.

To cahnge the RSS item title link, make sure you save the elements that contain the links into a field in your Dapp (for example: a "Read More" field). When you get your final RSS feed URL from Dapper, add this string to the end of it:

www.dapper.net/transform.php?dappName=...&extraArg_link=Read_More@href

Note that the Read_More part should contain the actual name of your field, case sensitive and with spaces replaced by underscores.