Getting ChatGPT working on Meta Smart Glasses via Messenger

UPDATE: There is a Part 2 to this story, that makes this work via Whatsapp and much less complicated, check it out here.

This is a fun little weekend project to get ChatGPT (kinda) working on the Meta Smart Glasses. I can image they will integrate this functionality very soon via Meta AI, but till then, we can have a little hacky fun. Big thanks to dcrebbin for their implementation which served as inspiration for this.

Guess who generated this image?

What it can do:

• Send a text query to ChatGPT and hear back the response
• Send an image & then a query connected to the image to ChatGPT and hear back the response

What you’ll need:

• Meta Glasses with Meta AI enabled
• OpenAI API key
• An alternative FB account (easy to set up), a FB app & and FB page
• Access to a hosted server (only for one step)
• A browser that can install web extensions and stay on

The main logic

Meta Glasses can send messages via voice to Messenger and Whatsapp contacts.

1. We’ll make a alternative FB profile that will receive the queries from our main FB profile (this must be a profile for now, not a page, since those are filtered out when looking for contact names in Meta Glasses).
2. We’ll make a browser extension that will monitor a browser window of the alternative FB profile for new messages, and process them via GPT.
3. We’ll make a new FB page and start a conversation with it from our main FB profile.
4. We’ll send the response back via GraphAPI to the FB page chat with out main FB profile, and Meta Glasses will read the notification back to us.

This is not ideal for many reasons, but it’s just a proof of concept and I can image others can improve on it.

1. Enable Meta AI on glasses

When setting up your glasses, if you’re outside the US, this will be disabled for you. You will need to un-pair the glasses, remove the Meta View app and install a VPN on your phone connected to the US. Then just re-setup the glasses as normal.

Note: During the setup process, after downloading the firmware update, the app may hang on an infinite connect loop “4 minutes remaining”. This happens because the VPN interferes with the transfer of the firmware to the glasses, it’s safe to disable it now, since it’s only needed during the firmware download.

After setup, connect Messenger via the Meta View app and enable reading messenger messages out loud in the settings.

2. Get an OpenAI key

This should be straight forward.

3. Set up an additional FB profile

FB lets you create multiple personal profiles and manage them from your main account. Just follow the official steps here: https://www.facebook.com/help/5637701596308695.

Name the profile something you can easily query and differentiate via the voice command. Then start a chat with it from your main FB Profile, so it will appear in your chats.

4. Set up a FB app, FB page & get the page-scoped user ID

In order to be able to send the response back to the user, we need some FB app & page logic.

• Create a new FB app, select Other -> Business as the app type during the setup.
• Add Messenger product to the app.
• From the Messenger API Setup page, in step 1, create a webhook to your server, just to be able to get a special page-scoped user ID. This is only needed during setup, and the server is not needed after.
  The best way to do this is to make a PHP script on a server you have handy (or any other language for that matter). Add the following code, host it somewhere, link to it in the Callback URL field, the Verify token is not important.

<?php
echo ($_GET['hub_challenge']);
$json = file_get_contents('php://input');
error_log($json);

• Once you have verified the Callback URL, subscribe the webhook to the messages field. For me, the Facebook GUI did not work in subscribing, so I had to use the Graph API for that.
• From the same screen, in step 2, create a new FB page that will send back the response to the user. Name it something understandable, for example <alternative profile name> Answer. Note: This is different from the additional FB profile that receives the requests.
• Finally, start a conversation with the new FB page from your main FB profile you have on your Meta Glasses (not the additional one). Once you send a message, your server you set up above will get the message, and you’ll be able to find your page-scoped user ID which is needed.
  Look in the error logs (or however you decided to output the webhook) for the id value in the sender object, note it down.

Note: There has to be a better way to get the page-scoped user ID, but no matter how much I tried, this was the only way that gave me a value that worked in later steps.

5. Get a non-expiring page-access token

In order to keep being able to send responses back to the user, we need a non-expiring page token. Just follow this tutorial to get.

6. Download and install the web extension

In order to know when the main FB profile sends a request to the alternative FB profile, we need to monitor that chat. FB does not have any public APIs to do this unfortunately, and the only way I found fit that does not mess with content policies is to use a browser extension.

You will have to have a chat window open in a browser window with the extension open. It will look for new incoming messages, parse them, send them to GPT, get the response, and send it to the FB page chat. It’s basically the brains of the operation.

Code for the web extension: https://github.com/jovanovski/meta-glasses-gpt/tree/main/messenger-approach

Download the files, and edit background.js to add:

• OpenAI key — from step 2
• FB page ID — from step 4
• FB page-access token — from step 4
• FB page-scoped user ID — from step 4

Once done, load the extension in the browser (preferably one you don’t use, or another profile, since you’ll need to be switched to the alternative FB profile in it):

• In Firefox: Open the about:debugging page, click the This Firefox option, click the Load Temporary Add-on button, then select any file in your extension’s directory.
• In Chrome: Open chrome://extensions/. At the top right, turn on Developer mode. Click Load unpacked. Find and select the app or extension folder.

Then load up Facebook in a tab, switch to your alternative profile, and open the chat that you started with your main FB profile (or start one if you haven’t). If all is well, you’ll get an alert after a few seconds saying that the extension observer has been initiated.

Keep this tab & browser open. If you close it, the connection will no longer work. Reopen it and enable the extension to restart the process.

7. Finally, try it out!

With a text prompt

• “Hey Meta, send a message on Messenger to <alternative FB profile>, say What is a good healthy breakfast idea?”
• Confirm, and wait for response.

With an image & prompt

• “Hey Meta, send a picture on Messenger to <alternative FB profile>”
• *Picture snapped and sent, profile confirms that image is received and awaiting next prompt for the image*
• “Hey Meta, send a message on Messenger to <altenrative FB profile> say What is on this picture?”
• Wait for response

Was this complicated as hell? Yes.

Will Meta add all of this functionality eventually? Yes.

Was it fun doing it? Yes.

Have fun hacking, and help me make this method better and easier!