New Google Analytics API / DataSource!

Posted by Felix Geisendörfer, on Dec 18, 2007 - in PHP & CakePHP » DataSources, Models & Behaviors

Hey folks,

sorry it took me forever, but after my old Google Analytics API fell apart due to the fact that Google published a new interface that also came with new reports / exporting formats I didn't have the time to come up with a new one.

Anyway, in a very productive CakePHP session with nate a couple weeks ago in Atlanta I finally implemented some new HttpSocket stuff that was needed (minimal support for cookies / ssl) for creating the new API as a datasource. Now that its done I'm fairly happy with the outcome of the new API : ).

Without further words: Download the 1.0 version here (A SVN:HEAD version of Cake 1.2 is highly recommended).

It doesn't have pretty API comments yet, so I'm gonna make up for it with a little tutorial:

Step 1: Set up your connection credentials

After you downloaded the API and placed it in app/models/datasources/google_analytics_source.php, open up your app/config.database.php and add the following lines:

php
  1. var $analytics = array(
  2.   'datasource' => 'google_analytics',
  3.   'user' => 'my-account@gmail.com',
  4.   'password' => 'my-password',
  5. );

Naming your connection $analytics is a choice, anything else will do as well.

Step 2: Set up a little test controller:

php
  1.  
  2. class AnalyticsController extends AppController{
  3.     var $uses = array();
  4.     var $Analytics = array();
  5.  
  6.     function beforeFilter() {
  7.         App::import('ConnectionManager');
  8.         $this->Analytics =& ConnectionManager::getDataSource('analytics');
  9.     }
  10.  
  11.     function list_profiles() {
  12.         // List all profiles associated with your account
  13.         $profiles = $this->Analytics->listSources();
  14.         debug($profiles);
  15.         exit;
  16.     }
  17.    
  18.     function show_reports() {
  19. // The quickest way to test if the API is working for you.
  20.         $report = $this->Analytics->report('Dashboard');
  21.         debug($report);
  22.        
  23. // Use a specific profile id (see list_profiles action above), do:
  24.         $report = $this->Analytics->report(array(
  25.             'profile' => '290723',
  26.             'report' => 'Dashboard'
  27.         ));
  28.         debug($report);
  29.  
  30. // You can also reference your profile by name instead
  31.         $report = $this->Analytics->report(array(
  32.             'profile' => 'www.thinkingphp.org',
  33.             'report' => 'Dashboard'
  34.         ));
  35.         debug($report);
  36.  
  37. // Retrieve the raw XML instead of an array to parse it yourself (lets say using SimpleXml in PHP5):
  38.         $report = $this->Analytics->report(array(
  39.             'profile' => 'www.thinkingphp.org',
  40.             'report' => 'Dashboard'
  41.         ), true);
  42.         debug($report);
  43.        
  44. // Retrieve a PDF report (make sure you set the right header before displaying):
  45.         $report = $this->Analytics->report(array(
  46.                 'profile' => 'www.thinkingphp.org',
  47.                 'report' => 'Dashboard',
  48.                 'format' => 'pdf'
  49.             ), true);
  50.         debug($report);
  51.  
  52. // Set some criteria on the report:
  53.         $report = $this->Analytics->report(array(
  54.                 'profile' => 'www.thinkingphp.org',
  55.                 'report' => 'Dashboard',
  56.                 'from' => '2007-12-17',
  57.                 'to' => '2007-12-18'
  58.             ));
  59.         debug($report);
  60.        
  61. /**
  62.  * Small Reference of options:
  63.  *
  64.  * profile: The id or name of the profile (optional, default = first profile)
  65.  * report: A report string like 'Dashboard', 'TrafficSources', 'Keywords' (optional, default = 'Dashboard')
  66.  * from: A yyyy-mm-dd formated date (optional, default = 1 month ago)
  67.  * to: A yyyy-mm-dd formated date (optional, default = today)
  68.  * tab: 0-4 (optional, default = 0)
  69.  * view: 0-9+ (optional, default = 0)
  70.  * format: 0-3 or 'pdf', 'xml', 'csv', 'tsv' (optional, default = 'xml')
  71.  * compute: 'average', other options unknown? (optional, default = 'average')
  72.  * query: Here you can manually pass (and overwrite) any part of the raw HTTP GET query made to Google Analytics. (optional, default = array())
  73.  */
  74.     }
  75. }
  76.  

Now you might wonder how get the list of reports that are available. Well usually that is pretty simple, just check the name of the menu entry you have open when browsing through the google analytics web interface and try using it in CamelCase. For example the 'Search Engines' view under traffic sources is the same as the 'SearchEngines' report. If this does not work, then use Firebug or something else to find the destination of one of the export links on the screen you are looking at. For example on the 'Direct Traffic' screen (under traffic sources as well), the xml export link points to 'export?fmt=1&id=5879334&pdr=20071116-20071216&cmp=average&&rpt=DirectSourcesReport' which means that the name of the report you need for this API is 'DirectSources'. If I get around I might create some listReports() function, but for now this should do the trick.

Another tip: If you are overwhelmed by the information returned by the default xml reports, try using the CSV / TSV ones, they should usually contain less unneeded information, but you'll have to parse them yourself.

Ok, I hope you guys enjoy and let me know if you have any questions / suggestions / problems.

-- Felix Geisendörfer aka the_undefined

Note: This data source uses web scraping. This means that Google could easily break it by making changes to the infrastructure of Google Analytics again. I also haven't heard back from google about their legal views on this technique since Late May 2006, but so far I haven't heard about Google approaching any vendors providing similar access to their service using web scraping either, which is a good sign.

Print this Post | Digg This | Stumble It | Delicious

58 Comments

[...] IMPORTANT: The stuff found in this article is outdated and not working anymore. Please see: New Google Analytics API / DataSource! for a newer version! [...]

Lucian Lature on Dec 18, 2007:

Nice job!...:)

Pacifists on Dec 18, 2007:

Heh the last week I was working on your old API ir remaking it to fit the changes that google introduced, just didn't have time to finish it. I was going to ask you to look at it when I'm finished, but you beat me to it, so I guess all I have to do now - is to use the new one :)

Thanks for the great job. I like the clarity of your code.

oli on Dec 18, 2007:

coole sache.

Juan Basso on Dec 18, 2007:

Very nice!

Someone to change in your datasource: You are using "uses('Xml');", replace it by "App::import('Core', 'Xml');". :)

Pacifists on Dec 18, 2007:

Juan: I've tried changing that and then 'AppHelper' not found and some other errors poped out. Using SVN cake 1.2, so maybe it's not the best thing to do :)

Matt on Dec 18, 2007:

Great stuff Felix!

Glad you're back.

Juan Basso on Dec 18, 2007:

Sorry. I write wrong.

The correct is: App::import('Xml', 'Core');

Snowcore on Dec 19, 2007:

Thanks for Datasource!
It's really powerful!

PHPDeveloper.org on Dec 19, 2007:

Felix Geisendorfer's Blog: New Google Analytics API / DataSource!...

Due to the new advancements Google has made in their Analytics ......

[...] Due to the new advancements Google has made in their Analytics API (Datasource) functionality, the API changed slightly and the code that Felix Geisendorfer had created to work with a CakePHP install wasn’t working anymore. He’s corrected the problem with this new post with the new code and an example of its use. In a very productive CakePHP session with nate a couple weeks ago in Atlanta I finally implemented some new HttpSocket stuff that was needed (minimal support for cookies / ssl) for creating the new API as a datasource. Now that its done I’m fairly happy with the outcome of the new API. [...]

[...] Due to the new advancements Google has made in their Analytics API (Datasource) functionality, the API changed slightly and the code that Felix Geisendorfer had created to work with a CakePHP install wasn’t working anymore. He’s corrected the problem with this new post with the new code and an example of its use. In a very productive CakePHP session with nate a couple weeks ago in Atlanta I finally implemented some new HttpSocket stuff that was needed (minimal support for cookies / ssl) for creating the new API as a datasource. Now that its done I’m fairly happy with the outcome of the new API. [...]

[...] Due to the new advancements Google has made in their Analytics API (Datasource) functionality, the API changed slightly and the code that Felix Geisendorfer had created to work with a CakePHP install wasn’t working anymore. He’s corrected the problem with this new post with the new code and an example of its use. In a very productive CakePHP session with nate a couple weeks ago in Atlanta I finally implemented some new HttpSocket stuff that was needed (minimal support for cookies / ssl) for creating the new API as a datasource. Now that its done I’m fairly happy with the outcome of the new API. [...]

Jeanviet on Dec 22, 2007:

Thanks for the update... still so hard for me to know exactly what i have to do with cake php. can you give us more details (or someone who comments as well)... can you explain all the steps we have to deal with cake php... or maybe just what have changed with the old method (do we need to install cake php 1.2 ? your code works with the old version ?)

[...] http://www.thinkingphp.org/2007/12/18/new-google-analytics-api/ addthis_url = 'http%3A%2F%2Fdavidwalsh.name%2Fgood-web-design-css-working-group-body-id-ie8-acid2-domassistant-25-google-analytics-wii-opera-sdk%2F'; addthis_title = 'Weekend+Links+-+Good+Web+Design%2C+CSS+Working+Group%2C+Body+ID%2C+IE8+Acid2%2C+DOMAssistant+2.5%2C+Google+Analytics%2C+Wii+Opera+SDK'; addthis_pub = ''; [...]

Felix Geisendörfer on Dec 27, 2007:

Jeanviet: You will need to check out CakePHP 1.2 from SVN and then follow the steps I provided above. Any older cake version will not work. If you have problems with this try to check out the CakePHP mailing list at google groups or hop on freenode IRC #cakephp - there are plenty people who will help you if you ask nice and politely.

But as always, see how much some google research may already help you.

[...] The dashboard is customizable, but the only way to integrate your stats into another product or a centralized dashboard for all your sites is via a web-scrapping API (for CakePHP no less). [...]

jeanviet on Dec 29, 2007:

Felix: everything is OK now, i put the new version 1.2...

Here you can see my beautiful stats:
http://jeanviet.info/stats/analytics/janusiens

Thanks once again Felix...

Maybe a little question that i was enable to solve.. i'm not an expert in php...

In google analytics datas, you have variables with numbers which are string (7) and not integer...
Example: $pageviews = "154 789"

I would like to change this to integer...

if i do this (int)$pageviews... i've got this: 154

how can i have this: 154789 with a integer variable
i try this (before int function): str_replace (' ', '', $pageviews)

But it doesn't work. the space doesn't disappear.

[...] New Google Analytics API / DataSource! (tags: php) [...]

Jeanviet on Jan 06, 2008:

Happy new year for those who read the comments... maybe 1 day, i'll have the chance to have an answer to my "spaces" problems in numbers.

Felix Geisendörfer on Jan 06, 2008:

Jeanviet:

I didn't see any problem with your code, however, I assume this is what happened. Your code looked like this:

str_replace ('', '', $pageviews)
$pageviews = (int)$pageviews;

This does not work b/c str_replace does not work on a reference to the variable you pass in, but on a copy of the value. This means you got to write something like this:

$pageviews = str_replace ('', '', $pageviews);
$pageviews = (int)$pageviews;

or shorter:
$pageviews = (int)str_replace ('', '', $pageviews);

Anyway, its a little inappropriate to ask general PHP questions on here, there are more then enough mailing lists and irc channels full of people to help you with those out there. But no worries, I hope this helps and you're using this API for something fun ; ).

David Lynes on Jan 09, 2008:

I am trying to run the same above and have installed as per the instruction but I am getting the following error messages.

Warning (2): preg_split() [function.preg-split]: Compilation failed: lookbehind assertion is not fixed length at offset 9 [CORE/cake/libs/http_socket.php, line 832]

I am using PHP5 with the CakePHP 1.2.0.6311 beta.

Please could anyone help or point me in the right direction please.

Timmy on Jan 09, 2008:

Nice work. I'm going to test this out today. I was curious though about what/if anything doesn't work if not running off the HEAD 1.2.

Thanks again.

Google Analytics API on Jan 11, 2008:

[...] Update! A great update to a version that works! Check out the most recent Google Analytics API! Tags:Analytics, API [...]

aleron on Jan 11, 2008:

great work

santi on Jan 29, 2008:

hi all could some one show me please some code for the viewer, or some steps to test it, thanks. i think i have it working but havent been able to see any data :S

Suchi on Jan 30, 2008:

Hello - I was trying this API out - but am facing with some problems. It seems that the lgin is not working fine for me.
I tried to echo the response we get in the login function - but that forwards me to another login box - with the correct user filled in, no password filled in and a Captcha to be filled in.
When i tried to get the same respone echoed (but now wrapped in htmlentities function) i get the following

Loading... Click here to continue Please click here to continue.

Can you pleae help me out with this.

Suchi on Jan 30, 2008:

well - now it seems that the login is working - but still i am not able to get what i want.
When i go to
http://path-to-my-cake/analytics/list_profiles/
I get a lot of warnings etc. and then nothing.
I turned up the debug level and also put in some echos in the listSources function.
When i echoed $response there - i get the login screen of google (naturally the code after that doesnt work ans it is not able to get the matching string)
For more troubleshooting - i echoed $this->connected() & $this->login() and both of them are 1. So why s this asking me to login??

Any idea what can be wrong?

Thank for any help
Suchi

Suchi on Jan 30, 2008:

PS: sorry i cant give u the URL of my app - as it is behind htaccess authentication.

Felix Geisendörfer on Jan 31, 2008:

Suchi: My guess is you are using an old cake version before I added simple cookie support to the HttpSocket.

Suchi Garg on Jan 31, 2008:

i downloaded cake_1.2.0.6311-beta.tar.gz and installed that afresh - so it shouldnt be that.

Illundalp on Feb 05, 2008:
CepOrelpunege on Feb 07, 2008:

If your different and a bit of a trend setter, not a trend follower, either go with the Licence Plate handbags or design and make your own. ford f 150 air bag light stays on

hfycf36rwb on Feb 07, 2008:

doripke6tc7 http://www.767196.com/914472.html qm5g9h0k2fjb1uca

Shyama Kant on Feb 09, 2008:

sir,
For Google Analytics api i have followed the steps written in http://www.thinkingphp.org/2006/06/19/google-analytics-php-api-cakephp-model/ as well as visited this url also http://www.thinkingphp.org/2007/12/18/new-google-analytics-api/

Please let me know which to follow and the required steps to implement the google analytics. Its important.

[...] К сожалению, для Google Analytics нет документированного API, поэтому информацию можно получить только экспериментальным путем или рассматривая соответствующие разработки. Я остановился на втором, за основу были взяты API Джо Тана2 для плагина WordPress Reports3, и Google Analytics API для CakePHP4. Второе не подходит в качестве базового решения, поскольку требует CakePHP знакомиться с которым пока желания нет, да и ставить фреймворк из-за простенького скрипта захочется не всем. В API от Джо я значительно упростил часть работающую непосредственно с GA, убрал лишние запросы и парсинг, которому в элементе отвечающем за загрузку делать нечего. Обратите внимание, что для авторизации требуются логин и пароль Google аккаунта, хранить на хостинге данные основного аккаунта не всегда целесообразно и безопасно, возможно стоит создать аккаунт специально для таких целей и открыть на нем доступ к отчетам сайта. [...]

Cees on Feb 18, 2008:

How do I download the whole stuff? The downloads on cakeforge.org all seem to point to some "download.php", while in the blog there is spoken of folders like "app/"/

Felix Geisendörfer on Feb 18, 2008:

Cees: You will need to download CakePHP 1.2 from cakephp.org first before you're going to be able to make any use of this. At this point there is no standalone version of this script.

WyriHaximus on Feb 23, 2008:

I keep getting this when I try to use this DS " We cannot log you into your account at this time. Please try again later.". Anyone knows how to fix/solve it?

avm on Mar 06, 2008:

I'm getting this error

Class 'App' not found in C:\wamp\www\cake\app\controllers\analytics_controller.php on line 22

The line involved is

App::import('ConnectionManager');

What is causing this? Help is appreciated

Felix Geisendörfer on Mar 06, 2008:

avm: You need to use the latest version of CakePHP 1.2 for this data source to work. You are probably using 1.1.

Jon Peltier on Apr 12, 2008:

I can't seem to avoid this error.

Fatal error: Class 'GoogleAnalyticsSource' not found in /www/eh7701/public_html/cake_1_2/cake/libs/model/connection_manager.php on line 110

Unfortunately I know little of PHP, though I'm picking it up fast.

Daniel.Interklub on Apr 15, 2008:

Hi!

Is there new version of method that allows to grab all Top Content section reports?
I need to get about 8000 positions and max passed via query is 500.

Theres no paginationations, so I dont know how to switch to next results.
I'll be very greatful for any suggestions.

DNA

Narendra on May 13, 2008:

Help!

My post looks like this (omitted email and pass) and I keep getting just another login page as the response. :-(

HttpSocket Object
(
[description] => HTTP-based DataSource Interface
[quirksMode] =>
[request] => Array
(
[method] => POST
[uri] => Array
(
[scheme] => https
[host] => www.google.com
[port] => 443
[user] =>
[pass] =>
[path] => /accounts/ServiceLoginBoxAuth
[query] => Array
(
)

[fragment] =>
)

[auth] => Array
(
[method] => basic
[user] =>
[pass] =>
)

[version] => 1.1
[body] => continue=http%3A%2F%2Fwww.google.com%2Fanalytics%2Fhome%2F%3Fet%3Dreset%26hl%3Den-US&service=analytics&nui=hidden&hl=en-US&GA3T=ouVrvynQwUs&Email=EEEEEE%40gmail.com&PersistentCookie=yes&Passwd=PPPPPPP
[line] => POST /accounts/ServiceLoginBoxAuth HTTP/1.1

[header] => Host: www.google.com
Connection: close
User-Agent: CakePHP
Content-Type: application/x-www-form-urlencoded
Content-Length: 232

[raw] => POST /accounts/ServiceLoginBoxAuth HTTP/1.1
Host: www.google.com
Connection: close
User-Agent: CakePHP
Content-Type: application/x-www-form-urlencoded
Content-Length: 232

continue=http%3A%2F%2Fwww.google.com%2Fanalytics%2Fhome%2F%3Fet%3Dreset%26hl%3Den-US&service=analytics&nui=hidden&hl=en-US&GA3T=ouVrvynQwUs&Email=EEEEE%40gmail.com&PersistentCookie=yes&Passwd=PPPPPP
[cookies] => Array
(
)

)

[response] => Array
(
[raw] => Array
(
[status-line] =>
[header] =>
[body] =>
[response] =>
)

[status] => Array
(
[http-version] =>
[code] =>
[reason-phrase] =>
)

[header] => Array
(
)

[body] =>
[cookies] => Array
(
)

)

[config] => Array
(
[persistent] =>
[host] => www.google.com
[protocol] => 6
[port] => 443
[timeout] => 30
[request] => Array
(
[uri] => Array
(
[scheme] => https
[host] => www.google.com
[port] => 443
)

[auth] => Array
(
[method] => basic
[user] =>
[pass] =>
)

[cookies] => Array
(
)

)

)

[lineBreak] =>

[_baseConfig] => Array
(
[persistent] =>
[host] => localhost
[protocol] => tcp
[port] => 80
[timeout] => 30
)

[connection] =>
[connected] =>
[error] => Array
(
)

[_log] =>
)

Felix Geisendörfer on May 13, 2008:

Narendra: Make sure you are using the latest version of CakePHP!

Michael on May 15, 2008:

Good job! Congratulations.
Thanks for share this app.

TimC on May 17, 2008:

Felix - thanks for sharing... One question from a Cake newbie. The API seems to work for me, but instead of a report or something, I get the following messages:

app/controllers/analytics_controller.php (line 29)
Error: The view for AnalyticsController::show_reports() was not found.
Error: Confirm you have created the file: app/views/analytics/show_reports.ctp

I understand what it is looking for, but have no idea what goes into the file!!

Any help appreciated.

Thanks in advance...

Kiyong on May 19, 2008:

Thanks~ Felix... :)
I'm Koeran.
You are lucky guy!
I'm happy for you.

Felix Geisendörfer on May 19, 2008:

TimC: Just create the file and put some text in it. You can also add PHP code to it and output any variables set from your controller using the Controller::set function.

TimC on May 26, 2008:

Felix - one more question. I have the ctp file created now, but noticed the debug($report) line in the control file was commented out. When I uncomment the debug($report) line, I get the following:

app/controllers/analytics_controller.php (line 29)

Line 29 is: debug($report);

here is the code in show_reports():

$report = $this->Analytics->report(array(
'profile' => '6723610',
'report' => 'Dashboard'
));
debug($report);

Similar error output occurs for the list_profiles and show_reports. The analytics page in Google has multiple accounts, but the specific id should take care of this (I believe). Just noted the multiple accts as an FYI. Also, running this on a Mac Book Pro w/ Safari and Firefox browsers.

Thanks again in advance...

Tim

TimC on May 26, 2008:

Felix - forget my previous request. I just reviewed my database.php file (for like the 10th time) and noticed a typo in my user name that I hadn't noticed before. Arrrghhh, but the good news is, it's working!!

Thanks again!!

Sean Neilan on May 30, 2008:

I'm getting some kind of SSL error. It says,
Warning (2): fread()function.fread]: SSL: fatal protocol error [CORE/cake/libs/socket.php, line 233

Running PHP 5.2.3 with Zend engine v2.2.0, Zend extension manager v1.2.0 and Zend optimizer v3.2.8

Any ideas?

Thank you very much for your time.

Felix Geisendörfer on May 30, 2008:

Sean: OS? I know osx can be a tiny bit quirky at times but you should not get this on linux.

Sean Neilan on May 30, 2008:

@Felix

Thank you for your response!

The PHP script is running server side on a redhat linux box @ hostforweb.com. Although, I'm running OSX here. Normally, I wouldn't think the client side would have much to do with an error like this.

Actually, for some reason, I didn't get this error before and when I showed it to my boss, it came up. Kind of like Michigan J. Frog.

Felix Geisendörfer on Jun 01, 2008:

Sean: Maybe you need to update your PHP version or SSL bindings. The HttpSocket itself relies on PHPs built-in magic for SSL so I wouldn't really know what could be causing this other then a buggy implementation somewhere down the stack.

narendra on Jun 18, 2008:

Hi Felix,

I tried a lot with your script, but still i am not getting any output.

its show me below errors

app\controllers\analytics_controller.php (line 21)

app\controllers\analytics_controller.php (line 28)

app\controllers\analytics_controller.php (line 35)

app\controllers\analytics_controller.php (line 42)

app\controllers\analytics_controller.php (line 50)

app\controllers\analytics_controller.php (line 59)

Notice (8): Undefined property: View::$Analytics [APP\controllers\analytics_controller.php, line 20]

Code

function show_reports() {

// The quickest way to test if the API is working for you.

$report = $this->Analytics->report('Dashboard');

AnalyticsController::show_reports() - APP\controllers\analytics_controller.php, line 20
include - APP\views\analytics\show_reports.ctp, line 2
View::_render() - CORE\cake\libs\view\view.php, line 646
View::render() - CORE\cake\libs\view\view.php, line 368
Controller::render() - CORE\cake\libs\controller\controller.php, line 732
Dispatcher::_invoke() - CORE\cake\dispatcher.php, line 260
Dispatcher::dispatch() - CORE\cake\dispatcher.php, line 230
[main] - APP\webroot\index.php, line 84

Fatal error: Call to a member function report() on a non-object in C:\xampp\htdocs\cakephp\app\controllers\analytics_controller.php on line 20

Please help me ,

Thanks

Alex on Jun 26, 2008:

Great script! Thanks Felix!

Has someone solved the problem of getting the larger datasets(more then 500 records)?

Felix Geisendörfer on Jun 26, 2008:

Alex: I'll look into it now.

Add a comment