Building apps with GPT-3? Here's what devs need to know about cost and performance

Asia's Tech News Daily

Microsoft, OpenAI, GPT-3, Follow @neural

Last week, OpenAI removed the waitlist for the application programming interface to GPT-3, its flagship language model. Now, any developer who meets the conditions for using the OpenAI API can apply and start integrating GPT-3 into their applications.

Since the beta release of GPT-3, developers have built hundreds of applications on top of the language model. But presents unique challenges. You must find a way to leverage the power of OpenAI’s advanced deep learning models to provide the best value to your users while keeping your operations .

Fortunately, OpenAI provides a variety of options that can help you make the best use of your money when using GPT-3. Here’s what the people who have been developing applications with GPT-3 have to say about best practices.

Models and tokens

Microsoft, OpenAI, GPT-3, Follow @neural
OpenAI provides GPT-3 in different sizes, prices, and performance levels.

OpenAI offers : Ada, Babbage, Curie, and Davinci. Ada is the fastest, least expensive, and lowest-performing model. Davinci is the slowest, most expensive, and highest performing. Babbage and Curie are in-between the two extremes.

OpenAI’s website doesn’t provide architectural details on each of the models, but includes a list of different versions of the language model. The main difference between the models is the number of parameters and layers, going from 12 layers and 125 million parameters to 96 layers and 175 billion parameters. Adding layers and parameters improves the model’s learning capacity but also increases the processing time and costs.

Microsoft, OpenAI, GPT-3, Follow @neural

OpenAI calculates the pricing of its models based on tokens. According to OpenAI, “one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words).”

Here’s an example from :

Microsoft, OpenAI, GPT-3, Follow @neural
Example of tokenized text

In general, if you use good English (avoid jargon, use simple words with few syllables, etc.), you’ll get better token-to-word ratios. In the example below, aside from “GPT-3,” every other word counts as one token.

Microsoft, OpenAI, GPT-3, Follow @neural

One of the benefits of GPT-3 is its . If you’re not satisfied with the model’s response to a prompt, you can guide it by giving it a longer prompt that includes correct examples. These examples will work like real-time training and improve GPT-3’s results without the need to readjust its parameters.

It is worth noting that OpenAI charges you for the total tokens in your input prompt plus the output tokens GPT-3 returns. Therefore, long prompts with few-shot learning examples will increase the cost of using GPT-3.

Which model should you use?

With a 75x cost difference between the cheapest and most expensive GPT-3 models, it is important to know which option best suits your application.

Matt Shumer, the co-founder and CEO of OthersideAI, has used GPT-3 to develop AI-powered writing tools. , OthersideAI’s main product, uses GPT-3 for text generation, autocomplete, and rephrasing.

When choosing between different GPT-3 models, Shumer starts by considering the complexity of the intended use case, he told TechTalks.

“If it’s something simple, like binary classification, I might start with Ada or Babbage. If it’s something very complex, like conditional generation where high-quality output and reliability is necessary, I start with Davinci,” he said.

When unsure of complexity, Shumer starts by trying the biggest model, Davinci. Then, he works his way down toward the smaller models.

“When I get it working with Davinci, I try to modify the prompt to use Curie. This typically means adding more examples, refining the structure, or both. If it works on Curie, I move to Babbage, then Ada,” he said.

For some applications, he uses a multi-step system that includes a mix of different models.

“For example, if it’s a generative task that requires some classification as a precursor step, I might use Babbage for the classification, then Curie or Davinci for the generative step,” he said. “After using it for a while, you get a feel for what might be useful for different use cases.”

Microsoft, OpenAI, GPT-3, Follow @neural
OpenAI’s Playground lets you directly try prompts on different GPT-3 models

Paul Bellow, author and developer of , used Davinci for his GPT-3-powered RPG content generator.

“I wanted to generate the highest quality output possible—for later fine-tuning,” Bellow told TechTalks. “Davinci is the slowest and most expensive, but the tradeoff is higher quality output which was important to me at this stage of development. I’ve spent a premium, but I now have over 10,000 generations that I can use for future fine-tuning. Datasets have value.” (More on fine-tuning later.)

Bellow says that the best way to find out if another model is going to work for a task is to run some tests on Playground, a tool you can use to directly try prompts on different GPT-3 models (note that OpenAI bills you for using Playground).

“A lot of the time, a well-thought-out prompt can get good content out of the Curie model. It all just depends on the use-case,” Bellow said.

Balancing costs and quality

When choosing a model for your application, you’ll have to weigh the balance between the cost and value. Choosing a high-performing model might provide better quality output, but the improved results might not justify the price difference.

“You have to build a business model around your product that supports the engines you’re using,” Shumer said. “If you want high-quality outputs for your users, it’ll be worth it to use Davinci—you can pass off the costs to your users. If you’re looking to build a large-scale free product, and your users are okay with mediocre results, you can use a smaller engine. It all depends on your product goals.”

OthersideAI has developed a solution that uses a mix of different GPT-3 models to enable different use cases, Shumer said. Paid users enjoy the power of large GPT-3 models, while free-tier users get access to the smaller models.

For LitRPG Adventures, quality is prime, which is why Bellow initially stuck to the Davinci model. He used the base Davinci model with one- or two-shot prompts, which increased the costs but made sure GPT-3 provided quality output.

“OpenAI API Davinci model is a bit expensive at this time, but I see the cost going down eventually,” he said. “What provides flexibility right now is the ability to fine-tune the Curie and lower models, or Davinci with permission. This will bring my costs per generation down quite a bit while hopefully maintaining high quality.”

He has been able to develop a business model that maintains a profit margin while using Davinci.

“While not a huge money-maker, the LitRPG Adventures project is paying for itself and just about ready to scale up,” he said.

Finetuning GPT-3

OpenAI’s scientists initially introduced GPT-3 as a task-agnostic language model. According to their initial tests, GPT-3 rivaled state-of-the-art models on specific tasks without the need for further training. But they also mentioned fine-tuning as a “promising direction of future work.”

In the months that followed the beta release of GPT-3, OpenAI and Microsoft fine-tuned the model for a number of different tasks, including and .

, fine-tuning has several benefits for GPT-3. OpenAI API allows customers to create fine-tuned versions of its GPT-3 for a premium. You can create your own training dataset, upload it to OpenAI’s servers, and use it to create a finetuned model of GPT-3. OpenAI will host your model and make it available to you through its API.

Fine-tuning will enable you to tackle problems that are impossible to solve with the basic models.

“The vanilla models are highly capable and are usable for many tasks. However, some tasks (i.e., multi-step generation) are too complex for a vanilla model, even Davinci, to complete with high accuracy,” Shumer said. “In cases like this, you have two options: 1) create a prompt chain that feeds outputs from one prompt into another prompt, or 2) fine-tune a model. I typically first try to create a prompt chain, and if that doesn’t work, I then move to fine-tuning.”

If done properly, fine-tuning can also reduce the costs of using GPT-3. If you’ll be using GPT-3 for a specific application, a fine-tuned small model can produce results that are as good as those provided by a large vanilla model. Fine-tuned models also reduce the size of prompts, which further slashes your token usage.

“One other case where I tend to fine-tune is when I can get something working with a vanilla model, but the prompt ends up being so long that it is costly to serve to users. In cases like these, I fine-tune, as it actually can reduce the overall serving costs,” Shumer said.

But fine-tuning isn’t without challenges. Without a quality training dataset, finetuning can have adverse effects.

“Clean your dataset as much as you can. Garbage in, garbage out is one of my big mantras now when it comes to prompt engineering,” Bellow said.

If you manage to gather a sizeable dataset of quality examples, however, fine-tuning can do wonders. After starting LitRPG with the Davinci model, Bellow gathered and cleaned a dataset of around 4,000 samples in a 7-megabyte JSON file. While he is still experimenting, the initial results show that he can move from Davinci to Curie without a noticeable change in quality, which reduces the costs of GPT-3 queries by 90 percent.

Another consideration is the time it takes to fine-tune GPT-3, which grows with the size of the model and the training dataset.

“It can take as little as five minutes to fine-tune a smaller model on a few hundred examples,” Shumer said. “I’ve also seen cases where it takes upwards of five hours to train a larger model on thousands of examples.”

There’s also an inverse correlation between the size of the model and the amount of data you need to fine-tune GPT-3, according to Shumer’s experiments. Larger models require less data for fine-tuning.

“For many tasks, you can think of increasing base model size as a way to reduce how much data you’ll need to fine-tune a quality model,” Shumer said. “A Curie fine-tuned on 100 examples may have similar results to a Babbage fine-tuned on 2,000 examples. The larger models can do remarkable things with very little data.”

GPT-3 alternatives

Microsoft, OpenAI, GPT-3, Follow @neural

OpenAI received a lot of criticism for deciding . Subsequently, other developers released GPT-3 alternatives and made them available to the public. One very popular project is . Like other open-source projects, GPT-J requires technical effort on the part of application developers to set up and run. It also doesn’t benefit from the ease of use and scalability that comes with hosting and fine-tuning your models on Microsoft’s Azure cloud.

But open-source models are nonetheless useful and are worth considering if you have the in-house talent to set them up and they meet your application’s requirements.

“GPT-J isn’t the same as full-scale GPT-3—but it is useful if you know how to work with it. It’s exponentially harder to get a complex prompt working on GPT-J, as compared with Davinci, but it is possible for most use-cases,” Shumer said. “You won’t get the same super high-quality output, but you can likely get to something passable with some time and effort. Plus, these models can be cheaper to run, which is a big plus, considering the cost of Davinci. We have successfully used models like these at Otherside.”

“In my experience, they operate at about the level of the Curie model from OpenAI,” Bellow said. “I’ve also been looking into , but they’re not giving details on the size of their model, so I imagine it’s around the same as GPT-J, et al. I do think (hope) that there will be even more options soon from other players. Competition between suppliers is good for consumers like me.”

This article was originally published by Ben Dickson on , a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original article .

Internet Explorer Channel Network
Asia's Tech News Daily
News Related


Why Is The Airline Industry So Concerned With The 5G Network?

While AT&T Inc (NYSE:T) and Verizon Communications Inc (NYSE:VZ) announced ambitious plans to deploy 5G services in the U.S., the airline industry has raised concerns about how this technology could ... Read more »

World faces shortage of lithium for electric vehicle batteries

Source: Reuters Lithium is in hot demand due to rapidly growing production of electric vehicles that use lithium-ion batteries, but there is a global supply shortage of the metal, with ... Read more »

Intel to build a $20-billion chip factory in Ohio

(Image Courtesy: Intel/TIME) Intel Corp is planning to massive a $20-billion chip manufacturing site in New Albany, Ohio in the US on a 1,000-acre site, as the world grapples with a ... Read more »

Raven Software QA workers unionize within Activision Blizzard

© Image: Infinity Ward/Activision Quality assurance workers at Raven Software, a subsidiary of Activision Blizzard, are unionizing with the Communication Workers of America (CWA). The group, called Game Workers Alliance, ... Read more »

Everything you need to know about Wi-Fi 7

As networking devices with Wi-Fi 6 are becoming widely available and relatively affordable, tech companies are already looking ahead to implement Wi-Fi 7 in their products and services. This new ... Read more »

Did Elon Musk forget about OpenAI or is he just trolling his dumbest fans?

It’s impossible to tell if Elon Musk is serious about anything anymore. His image exists in a dichotomy between cartoonesque and genius. He’s simultaneously the richest man in the world ... Read more »

CESL launches tender for 5,500 electric buses in 5 cities

Representative Image Convergence Energy Services (CESL), a wholly-owned subsidiary of Energy Efficiency Services (EESL), on January 20 announced a tender for 5,500 electric buses in five cities — Delhi, Kolkata, ... Read more »

Families forced to cancel half term holidays because of ‘restrictive' Covid pass rules

Children between the age of 12-15 are unable to prove their vaccine status on the NHS Covid app – JUSTIN TALLIS Families have been forced to cancel half term holidays ... Read more »

Redmi Note 11 Pro 5G design renders leaked ahead of January 26 global launch

Redmi Note 11 Pro 5G will feature a Snapdragon processor. Redmi Note 11 Pro design renders have leaked online ahead of the global launch on January 26 even as the ... Read more »

Garmin Fenix 7 too expensive? Check out the super light Suunto 5 Peak instead

If your pockets aren’t deep enough for the newly launched Garmin Fenix 7 multi-sports watch the Suunto 5 Peak is an alternative that’s well worth considering. The new watch is ... Read more »

Godzilla is getting an Apple TV show set in the Monsterverse

Spoilers follow for Godzilla vs Kong. Godzilla fans, rejoice: a live-action TV series, set in Legendary Pictures’ Monsterverse, is in development for Apple TV Plus. Announced in a press release, ... Read more »

Winning the 'Great Transition'

© Getty Images Job board It is likely that more people will quit their jobs – and find new ones – in the next 100 days than at any other ... Read more »

Exclusive | No Siri, Alexa, Smart Devices in Secret Meetings: Intel Note to Govt Officials on Tech Threats

(Image Courtesy: Reuters) Witnessing massive violation of national communication security policy guidelines and instructions of the government by officials and to control leaks of classified information, a new communication security ... Read more »

AT&T, Verizon begin turning on C-band 5G

Representative image After a six-week delay, and a whole lot of headaches, AT&T and Verizon have finally started to activate their C-Band 5G networks. People in some areas of the ... Read more »

Snapchat implements new safety features for teens

Snapchat has been busy trying to tackle a drug problem on its platform Snapchat is introducing new safety features for teens that limits the friend suggestions they get to people ... Read more »

Oppo Reno 7, Reno 7 Pro India launch confirmed via Flipkart; to feature World's first Sony IMX709 sensor

OPPO Reno 7 Pro will compete against the likes of the Xiaomi 11T Pro, OnePlus 9RT, etc. Oppo Reno 7 India launch has been confirmed. The company has said that its ... Read more »

FedEx Express commences electric vehicle trials in India

Source: Shutterstock FedEx Express, a subsidiary of leading global express transportation company FedEx Corp, on Friday announced the commencement of electric vehicle (EV) trials in India with Bengaluru to begin ... Read more »

Explainer | Artificial Intelligence, and its impact on advertising

Representational image. Srivibhu Timmaraju and K Rajeshwari “Artificial Intelligence is a tool, not a threat”: Rodney Brooks, Professor of Robotics, Massachusetts Institute of Technology Ever been in a situation where ... Read more »

China 3-4 generations away from ‘cutting-edge' chip capabilities: Report

File photo of a semiconductor chip (Image Source: Shutterstock) Mario Morales, group vice president for enabling technologies and semiconductors at the International Data Corporation (IMC) feels China is “generations” away ... Read more »

5G Ultra Wideband just appeared on my iPhone and it's the speed I've always wanted

474.51 Mbps. I had never seen a mobile download speed like it before. 5G Ultra Wideband – yes, the controversial 5G C-Band you’ve been reading about for months – is ... Read more »

Pre-order Samsung's The Freestyle projector and get a free gift - ending soon!

Upgrade your home cinema experience to a whole new level with this pre-order deal on Samsung’s The Freestyle projector. For a limited time, Samsung is offering a free carrying case ... Read more »

Vergecast: Microsoft to acquire Activision, Google building a headset, and the 5G battle with airlines

© Image: Blizzard Every Friday, The Verge publishes our flagship podcast, The Vergecast, where we discuss the week in tech news with the reporters and editors covering the biggest stories. ... Read more »

Los Angeles weighs phasing out oil and gas drilling

The University Park neighborhood of Los Angeles has a lot in common with urban areas across the U.S.: A dense population with lots of businesses and housing. A cluster of ... Read more »

Ashley Park Reveals 'Shock' From BTS Fangirl Moment Turned Out To Be COVID

Ashley Park has revealed she went into such a deep state of shock after the stars of BTS endorsed one of her performances that she didn’t realize she was actually ... Read more »

Sony seeking new partners for transformative EV project: Report

Sony Group will likely add new technology partners to its electric vehicle (EV) project to help it forge a mobility business to transform cars from transportation machines to entertainment spaces, ... Read more »

New York Mayor Adams to receive first paycheck in cryptocurrency

New York City Mayor Eric Adams on Thursday said his first paycheck, to arrive on Friday, will be automatically converted into cryptocurrency via Coinbase Global Inc. The salary will be ... Read more »

Epic Galaxy Fold 3 and Flip 3 deal: free Buds and $150 off Galaxy Watch 4 at Samsung

Samsung’s upping the ante this week with an array of awesome deals on its foldable devices – the Samsung Galaxy Z Fold 3 and Flip 3. Right now, not only ... Read more »

Apple implements new verification process for educational discounts in US

Tech giant Apple has introduced a new verification process in the US to ensure that customers who want to benefit from its discounted education pricing are involved in education.According to ... Read more »

Samsung Galaxy S22 reservation teaser confirms Note-like design for S22 Ultra

The Samsung Galaxy S22 launch is expected very soon, and ahead of the February Unpacked event, Samsung has launched a reservation page so you can get the company to set ... Read more »

Apple fires back in Ericsson 5G legal battle

Apple has countersued Ericsson and is seeking an important ban on the import of the Swedish telecoms equipment manufacturer’s base stations into the US as part of a long-running legal ... Read more »

Casio's CT-S1000V keyboard makes it easy to sound like Daft Punk

Casio made a name for itself providing lots of bang-for-your-buck with its musical instruments, as evidenced by my review of the excellent $200 CT-S1 last year. Now the company is ... Read more »

The best macOS apps for your brand new MacBook in 2022

Of all the great feelings in the world, one towers above all. That’s right, I’m talking about getting a new Apple computer. Besides it obviously being a shiny new thing, ... Read more »

Ethereum inventor wants to replace pregnant women with synthetic wombs

Crypto bros never tire of sharing their utopian visions, but their big brain ideas aren’t always embraced by the proles. Vitalik Buterin, the co-founder and inventor of Ethereum, has become ... Read more »

Rimac's selling its hypercar EV tech to other automakers

We’ve already been blown away by Rimac’s breathtaking Nevera, the all-electric hypercar that goes 0-60mph in just 1.85 seconds. And we were stunned when Rimac swallowed Bugatti to become, well, Bugatti ... Read more »

This dystopic-looking bot could scrub greenhouse gases from the air

Two centuries of burning fossil fuels has put more carbon dioxide, a powerful greenhouse gas, into the atmosphere than nature can remove. As that CO2 builds up, it near Earth’s ... Read more »

This car can (probably) mine crypto and we're not sure that's a good thing

The thing mobility and blockchain technology have in common are grandiose ideas that promise a lot, but don’t consistently deliver. So can a product that combines both overcome this? Canadian company, Daymak, ... Read more »

EVERYBODY PANIC! The Doomsday Clock is close to the hour of our self-annihilation

It’s been a tough couple of years, but things may soon get a lot worse: the Doomsday Clock just struck 100 seconds to midnight — the hour of humanity’s self-annihilation. ... Read more »

Google's G Suite is no longer free — here are the best alternatives

Google has done it again. After killing free unlimited storage on Google Photos last year, it has announced that G Suite will become a paid-only feature. As  reported, all free G ... Read more »

People no longer accept shitty jobs, and it's fuelling the Great Resignation

“Do what you love,” is no longer just advice. High school students learn early on that their future careers should be passion-driven. Self-help books counsel job searchers to start with ... Read more »

5G is grounding planes and freaking out airlines: We found out why

The journey to 5G is having a turbulent takeoff in the US. The rollout of the wireless service has been partially postponed due to fears that it poses a threat ... Read more »
On you will find lots of free English exam practice materials to help you improve your English skills: grammar, listening, reading, writing, ielts, toeic