Or, How to Turn a Jailbroken iPhone into Jarvis

Welcome to the future. The future where you can speak to your phone and have your light come on. Yeah really, we’re here, and the future’s just waiting for you.

Yup, this future.

What We’re Going to Do

We’re going to use Googiri in combination with wit.ai and IFTTT to automate the world around us.

We’ll start with a simple way for us to ask questions in a Slack channel. At the end of this, we’ll be able to ask Google to “tell everyone in general that it’s pizza time” and have Matt says: **it's pizza time** appear in your Slack channel.

Table of Contents

  1. The Tools We’re Using
  2. Getting and Testing Googiri
  3. Creating the Initial Version of the Webserver
  4. Integrating wit.ai
  5. Integrating IFTTT
  6. Success!

The final code is also available on GitHub.

Googiri, wit.ai, and IFTTT

Googiri

Googiri is a middleware broker for Google’s speech-to-text. Basically, you get to choose what happens after Google turns your speech into text. You can either let Google handle the search as normal, have it sent to Siri (for reminders or alarms), or have it sent to a url of your choosing as a GET request with a ?q= query parameter.

You need a jailbroken iDevice to use Googiri, of course. If you don’t have a jailbroken device, the information in this tutorial is still applicable, but you won’t be able to put the entire project together.

wit.ai

wit.ai is a natural language parsing service that was recently acquired by Facebook (most likely partially powering the M virtual assistant). After training your wit.ai model with example sentences, you can use the model as an API to parse new sentences. For example, it can take take out the trash and let you know that the user wants to take_something where something="trash".

IFTTT

IFTTT is an awesome platform that connects services using the simple premise of IF this THEN that. IFTTT supports the Maker Channel which is perfect for DIY solutions like ours. Simply send IFTTT an HTTP request to trigger an action on another service or configure a recipe to send your own service a webhook when something of interest happens. The possibilities are endless!

We’ll be using the IFTTT Maker Channel to post messages to Slack.

ngrok

To make our locally-hosted node.js app available over the internet, I’ll be using ngrok. ngrok makes http://localhost:8000 accessible via http://somesubdomain.ngrok.com, perfect for testing.

First Steps

Jailbreak your iDevice and download Googiri (from Cydia) and the Google Search App (from the App Store).

Lets test that Googiri works by having it send your request to Siri. Say to Google Siri, what time is it? and Siri should pop up to answer the question.

Awesome! Now lets go to Googiri’s settings page and configure the webserver address.

I’ve put in http://googiri.ngrok.com as the webserver address. Now, after saying something like Jarvis, what time is it to Google, Googiri will make a GET request to http://googiri.ngrok.com/?q=what+time+is+it, which will be processed by the server running locally on my computer.

The Webserver

Let’s create the webserver that will be processing our requests. We’re going to use node.js 4.0.0 and Express.

'use strict';

let express = require('express');
let app = express();

app.get('/', function (req, res) {

  console.log(req.query);

  res.send('Hello World!');
});

let server = app.listen(8000, function () {
  console.log(`listening on localhost:${server.address().port}`);
});

For now we’re just printing out the query parameters sent in the GET request. Don’t forget to install express with:

npm i --save express

Now you can run your server and test the connection:

$ googiri-demo git:(master) $ node index.js
listening on localhost:8000
{ q: 'what time is it', context: 'default' }

Awesome!

Hooking it up to wit.ai

Lets create a wit.ai account by signing up with GitHub and then create our first app.

Now you can train your app with some example sentences. I chose tell engineering that I'm deploying to master, among others.

Lets test the API provided by wit.ai.

It works!

You can see in the above screenshot that wit.ai understands that we want to announce_to_slack, but it doesn’t know how to parse the sentence just yet. Next, lets improve wit.ai’s model of our sentences by telling it which parts of the sentence we care about. We want the I'm deploying to master part of the sentence to be a message_body entity.

Once the model is all green like below, you’re ready to test again!

Awesome, we got wit.ai to parse the entities out of our example sentences to isolate the message body!

Add Some Code

Now lets hook Googiri up to wit.ai. For this, I’m going to use a promise-based HTTP request library called axios.

npm i --save axios
'use strict';

let axios = require('axios');
let express = require('express');
let app = express();

let WIT_AI_TOKEN = process.env.WIT_AI_TOKEN;

app.get('/', function (req, res) {

  // Now that we have the query,
  // lets ask wit to parse it and return some helpful information
  axios({
    url: 'https://api.wit.ai/message',
    method: 'GET',
    params: {
      v: '20160202',
      q: req.query.q,
    },
    headers: {
      'Authorization': `Bearer ${WIT_AI_TOKEN}`,
    },
  })
  .then(function(response) {
    // for now, just print the response data
    console.log(response.data);
  })
  .then(function() {
    // tell Googiri we're done by completing the request
    res.send('OK');
  })
  .catch(function(err) {
    console.error(err);
    res.send('Hello World!');
  });

});

let server = app.listen(8000, function () {
  console.log(`listening on localhost:${server.address().port}`);
});

Make sure to add WIT_AI_TOKEN to your environment with:

$ export WIT_AI_TOKEN=YOURWITAITOKENHERE

Let’s Add Some IFTTT

Now lets add IFTTT integration to post the message to Slack. We’ll be using the IFTTT Maker Channel.

Go ahead and start creating a new recipe using the Maker Channel’s Incoming Web Request as a trigger. You can name your event anything you’d like, but I chose incoming_slack_message.

Next, in the Action Ingredients, set {{Value2}} as the message content and {{Value1}} as the title content.

And let’s test it…

Add More Code

We’ll send another GET request, this time to IFTTT, with the contents of the data parsed by wit.ai. It’ll look a little something like this:

'use strict';

let axios = require('axios');
let express = require('express');
let app = express();

let WIT_AI_TOKEN = process.env.WIT_AI_TOKEN;
let IFTTT_MAKER_KEY = process.env.IFTTT_MAKER_KEY;

app.get('/', function (req, res) {

  // Now that we have the query,
  // lets ask wit to parse it and return some helpful information
  axios({
    url: 'https://api.wit.ai/message',
    method: 'GET',
    params: {
      v: '20160202',
      q: req.query.q,
    },
    headers: {
      'Authorization': `Bearer ${WIT_AI_TOKEN}`,
    },
  })
  .then(function(response) {
    // let's find the message we want to send
    let messageBody = response.data.outcomes[0].entities.message_body[0].value;
    console.log(`Announcing to slack: ${messageBody}`);
    return messageBody;
  })
  .then(function(message) {
    // lets tell IFTTT to post the message
    return axios.get(`https://maker.ifttt.com/trigger/incoming_slack_message/with/key/${IFTTT_MAKER_KEY}`, {
      params: {
        'value1': 'Matt wants to say:',
        'value2': message,
      },
    });
  })
  .then(function(response) {
    console.log(response.data);
    // the request was successful!
    res.json({
      title: 'Successfully Posted to Slack',
      text: 'You\'ve just hooked up Googiri, wit.ai, IFTTT, and Slack!',
    });
  })
  .catch(function(err) {
    console.error(err);
    res.send('OH NOES');
  });

});

let server = app.listen(8000, function () {
  console.log(`listening on localhost:${server.address().port}`);
});

Again, don’t forget to add IFTTT_MAKER_KEY to your environment with:

$ export IFTTT_MAKER_KEY=YOURIFTTTMAKERKEYHERE

Now let’s run the server and test the entire thing…

We Did It, We Did It, We Did It, YEAH

Awesome, you’ve now successfully hooked up Googiri, wit.ai, IFTTT, and Slack. You should have a solid foundation for innovating further. If you build something using this tutorial that you’re proud of, I’d love to know!

The final code is available on GitHub.

(this awesome gif brought to you by partyparrot)