I think you need to set the scope of the OAuth permissions to this value:
googleapis.com/auth/actions.fulfill
...
If you are basing your code in the login_with_google.php script that comes with the package, locate a line in that script that says:
$client->scope = 'https://www.googleapis.com/auth/userinfo.email '. 'https://www.googleapis.com/auth/userinfo.profile';
Change it to something like:
$client->scope = 'https://www.googleapis.com/auth/userinfo.email '. 'https://www.googleapis.com/auth/userinfo.profile' . 'https://www.googleapis.com/auth/actions.fulfillment.conversation
';
That should grant your application the permissions to send API calls to send push notification actions.