We not long ago experienced a shopper who is a multi-national retailer with each a physical and World wide web existence. The shopper necessary a way to acquire selected business enterprise intelligence (BI) details from the Online on a daily basis. Soon after numerous unsuccessful tries to create this functionality them selves, they came to us for a option.
On the surface the demands seemed to be tricky and it was uncomplicated to see why their own IT team had unsuccessful to uncover a resolution. They were being thinking “inside the box”, however, and hadn’t regarded as third-bash options. The requirements demanded that the application complete all of these duties:
Retrieve new merchandise listings on competitor’s web sites.
Retrieve existing pricing for all products shown on competitor’s world wide web internet sites.
Retrieve whole text of competitor’s Press Releases and public financial studies.
Keep track of all inbound backlinks pointing to competitor’s internet internet sites from other world-wide-web web sites.
At the time the data was acquired it required to be processed for reporting functions and then stored in the knowledge warehouse for long run entry.
Immediately after reviewing current net-centered details acquisition technological know-how, like “spiders” which crawled the World wide web and returned facts which then had to be processed as a result of HTML filters, we decided that the Google API and Internet Companies made available the greatest resolution.
The Google API gives distant accessibility to all of the search engine’s exposed functionality and presents a interaction layer which is accessed by means of the “Easy Item Entry Protocol” (Soap), a world wide web expert services conventional. Because Cleaning soap is an XML-based mostly technologies it is quickly built-in into legacy web-enabled applications.
The API met all of the necessities of the application in that it:
Delivered a methodology for querying the World wide web utilizing non-HTML interfaces
Enabled us to timetable normal look for requests developed to harvest new and up to date information and facts on the target topics.
It furnished data in a structure which was able to be conveniently integrated with the client’s legacy methods.
Working with the Google API, Soap and WSDL, our developers have been equipped to outline messages that fetched cached webpages, searched the Google document index and retrieve the responses without owning to filter out HTML or reformat the information. The resulting facts was then handed off to the client’s legacy programs for validation, reporting and further more processing ahead of achieving the info warehouse.
All through the Evidence of Notion period we ran tests wherever we ended up ready to reliably recognize and retrieve up-to-date public relations and investor relations info that exceeded the client’s anticipations.
In our following examination we retrieved the most at this time offered item internet pages which ended up listed in Google and then ran a further question to retrieve the Google “cached web page” variations. We ran these two information sets by way of variance filters and have been in a position to deliver precise price tag improve and lessen reviews as effectively as determine new merchandise.
For google index download used the Google API’s skill to obtain the “hyperlink:” attribute to fast build lists of inbound links.
These confined checks demonstrated that the Google API was able of manufacturing the BI information that the shopper requested as very well as demonstrating that the facts could be returned in a pre-outlined format which removed the require to use submit retrieval filters.
The shopper was delighted with the benefits of our Evidence of Concept period and authorized us to proceed with creating the resolution. The software is now in everyday use and is exceeding the client’s general performance anticipations by a broad margin.