ChatGPT can now access the Internet and execute the code it writes

Language model AIs teach themselves the arts of communication and problem solving based on a limited set of training data. In the case of GPT-4, that data is pretty stale, with the cutoff at the end of 2021. That’s where all of ChatGPT’s “knowledge” up to this point comes from, and its only output at least in the service the public can use is text status. Now, with today’s launch of an ecosystem of plugins, GPT levels up again with some impressive new abilities.

First of all, he now has access to the internet, which means he can surf the web for answers if he determines you need up-to-date information that isn’t in his knowledge base. To do this it formulates relevant search strings, submits them to search engines and databases like Bing, Google, GitHub and many others, reviews the results, then goes reading the links it deems worthy until it decides it has a good answer for you . You can see exactly what he’s doing as he does it, and when your answer comes back, it’s neatly annotated with links you can click to go and peruse the relevant sources yourself.

For now, your web browser activities are read-only other than sending “get” requests to select search engines and databases. It can’t fill out forms or do anything else online, so it can’t calmly go install unblocked copies of itself on some hidden server somewhere and start engaging in the kind of “power-seeking behavior” that has already been caught to exhibit.

Open AI

However, OpenAI keeps everything that happens within its search API separate from the rest of its infrastructure for security. It can’t visit websites that aren’t available through Bing’s “safe mode,” and it won’t visit sites that request not to be crawled in their robots.txt files.

Secondly, it can now execute the code it writes. OpenAI provided him with a working Python interpreter, sitting in a “firewalled, sandboxed execution environment,” along with some disk space, which remains available for the duration of the chat session or until it times out. It can now also upload and download files.

So if you ask it a question that requires some serious number crunching, it is now able to code a task-specific piece of software and run that code to complete your task. You can give it data in certain file formats and it will perform operations on that data and give you something back, potentially in a different format if that’s what you ask.

This is pretty crazy stuff. It will take a spreadsheet and create annotated graphs for you. It will accept the JPGs, tell you what they look like, and write and execute code to scale those images or convert them to grayscale.

Open AI

And it gets access to a number of initial third-party plug-ins, with many more to follow. For example, Expedia, OpenTable, and Kayak plugins can search and set reservations for flights, restaurants, accommodations, and rental cars. Instacart, Klarna and Shop plugins can find and compare products and set orders. A Wolfram|Alpha plug-in gives GPT access to mathematical and computational powers, as well as real-time data streams.

At this stage, it seems its capabilities are mostly limited to setting things up rather than actually transacting with your money; you will have to click and manage the money yourself.

Finally, a Zapier plugin acts as a gateway through which GPT can now access around 5,000 other apps, including Gmail, Google Sheets, Trello, HubSpot, and Salesforce. This begins to position GPT as the ultimate personal assistant, with access to a huge amount of personal and business information, and potentially permissions to log in and perform a variety of tasks for you. Extraordinary stuff.

These plugins are gradually becoming available to paid users and developers through a waiting list. And new plugins will proliferate at an amazing rate, since no one even needs to code them. “You write an OpenAPI manifest for your API, use human-language descriptions for everything, and you’re done,” developer Mitchell Hashimoto tweeted. “You let the model figure out how to authenticate, chain calls, process data in between, format it for display, etc. There’s absolutely no glue code.”

The pace of OpenAI progress has been absolutely breakneck in recent months. It looks like this insanely advanced AI is getting a massive overhaul with amazing new abilities every time we blink. These new plugins represent ChatGPT starting to break out of its cage and into the real world.

For now, its capabilities will be extremely limited, because OpenAI knows better than anyone the potential dangers of this unique technology. But assuming these guys are the good guys and have taken the time to make sure this happens safely, the massively disruptive aspect of GPT is sure to compel other, less principled and less capable players to rush to develop competing AIs and give them power competition.

The opportunities here are absolutely incredible and the risks are unprecedented with every step this technology takes. We’re in uncharted territory at this stage, with very limited forward vision and the accelerator pedal stuck to the floor. What a time to be alive.

Source: Open AI