Part 2: Getting ChatGPT working on Meta Smart Glasses via Whatsapp

Part 1 of this series involved using Facebook, now I’ve moved to a better method of using Whatsapp. Follow along:

What it can do:

• Send a text query to ChatGPT and hear back the response
• Send an image & then a query connected to the image to ChatGPT and hear back the response

What you’ll need:

• Meta Glasses with Meta AI enabled
• OpenAI API Key
• A Whatsapp account
• A web-facing server that can run code (PHP in this example)

The main logic

Meta Glasses can send messages via voice to Messenger and Whatsapp contacts.

1. We’ll make a developer Whatsapp bot that will receive messages from the glasses.
2. When each message is received, it will be sent to ChatGPT for processing.
3. We’ll send the response back via our Whastapp bot to our Whastapp main account, and Meta Glasses will read the notification back to us.

Step by step guide

1. Enable Meta AI on glasses

When setting up your glasses, if you’re outside the US, this will be disabled for you. You will need to un-pair the glasses, remove the Meta View app and install a VPN on your phone connected to the US. Then just re-setup the glasses as normal.

Note: During the setup process, after downloading the firmware update, the app may hang on an infinite connect loop “4 minutes remaining”. This happens because the VPN interferes with the transfer of the firmware to the glasses, it’s safe to disable it now, since it’s only needed during the firmware download.

After setup, connect Messenger via the Meta View app and enable reading messenger messages out loud in the settings.

2. Upload code to a server

• Copy these 2 files to a server that accepts PHP: https://github.com/jovanovski/meta-glasses-gpt/tree/main/whatsapp-approach
• Make sure you have composer installed.
• Run composer install in the directory of the files.
• Allow access to the folder where the file is located to www-data if using Apache or whatever user runs the server. This will let us create and delete files when needed.

3. Get an OpenAI key

This should be straight forward. Add the key to the OPENAI_KEY constant in the PHP script.

4. Create a Facebook app

• Via https://developers.facebook.com/, create a new app, select Other -> Business as the app type during the setup.
• Add Whatsapp product to the app.
• From the Quickstart -> Configuration section, set up the webhook to point to the full URL of the callback.php file you just set on your server.
• Once verified, subscribe to the messages webhook field.
• From the same page, in the Phone number section, click on Manage phone number to get your test phone number, and write it down in your contacts. This is the number you’ll be messaging.

5. Set up the script

• Uncomment the 2 debugging lines in the script (look around line 130).
• Send a test message on Whastapp to the new contact you just created in the previous step.
• If the permissions are set right, you’ll see a new file debug.txt show up in the same folder as the other scripts on the server. Look at the contents for 2 variables you need to add to your PHP script.
• Add the value from the key phone_number_id to the WHATSTAPP_SENDER_ID constant in the PHP. Add the value from to the WHATSAPP_INCOMING_PHONE_NUMBER contacts in the PHP script, this makes sure only messages from your number are being processed.
• Follow this guide to get a permanent Whatsapp access token, which you’ll insert into the WHATSAPP_TOKEN constant in the PHP script.
• Comment back the 2 debugging lines you uncommented in the first step of this section.

6. Test it all out

• You can use Whatsapp during testing, no need for the glasses yet. Send a message to the contact and see if you get a response back.
• You can also send an image to the contact, and it will await a query for the image that you need to send in a separate text after it.

7. Finally, try it out on the glasses

With a text prompt

• “Hey Meta, send a message on Whatsapp to <new contact name>, say What is a good healthy breakfast idea?”
• Confirm, and wait for response.

With an image & prompt

• “Hey Meta, send a picture on Whatsapp to <new contact name>”
• *Picture snapped and sent*
• “Hey Meta, send a message on Whatsapp to <new contacnt name> say What is on this picture?”
• Wait for response

Have fun hacking!