What AI researchers can learn from the NFL Combine

Artificial intelligence, Google, Follow @neural

How useful is your AI? It’s not a simple question to answer. If you’re trying to decide between Google‘s Translate service or Microsoft’s Translator, how do you know which one is better?

If you’re an AI developer, there’s a good chance you think the answer is: benchmarks. But that’s not the whole story.

The big idea

Benchmarks are necessary, important, and useful within the context of their own domain. If you’re trying to train an AI model to distinguish between cats and dogs in images, for example, it’s pretty useful to know how well it performs.

But since we can’t literally take our model out and use it to scan every single image of a cat and dog that’s ever existed or ever will exist, we have to sort of guess how good it will be at its job.

To do that, we use a benchmark. Basically, we grab a bunch of pictures of cats and dogs and we label them correctly. Then we hide the labels from the AI and ask it to tell us what’s in each image.

If it scores 9 out of 10, it’s 90% accurate. If we think 90% accurate is good enough, we can call our model successful. If not, we keep training and tweaking.

The big problem

How much money would you pay for an AI capable of telling a cat from a dog? A billion dollars? Half a nickel? Nothing? Probably nothing. It wouldn’t be very useful outside of the benchmark leaderboards.

However, an AI capable of labeling all the objects in any given image would be very useful.

But there’s no “universal benchmark” for labeling objects. We can only guess how good such an AI would be at its job. Just like we don’t have access to every image of a cat and dog in existence, we also cannot label everything that couldpossiblyexist in image form.

And that means any benchmark measuring how good an AI is at labeling images is an arbitrary one.

Is an AI that’s 43% accurate at labeling images from a billion categories better or worse than one that’s 71% accurate at labeling images from 28 million categories? Does it matter what the categories are?

BD Tech Talks’ Ben Dickson put it best in :

The focus on benchmark performance has brought a lot of attention to machine learning at the expense of other promising directions of research. Thanks to the growing availability of data and computational resources, many researchers find it easier to train very large neural networks on huge datasets to push the needle on a well-known benchmark rather than to experiment with alternative approaches.

We’re developing AI systems that are very good at passing tests, but they often fail to perform well in the real world.

The big solution

It turns out that guessing performance at scale isn’t a problem isolated to the world of AI. In 1982, National Football Scouting Inc. held the first “” to address the problems of busts – players who don’t perform as well as they were predicted to.

In the pre-internet era, the only way to evaluate players was in person and the travel expenses involved in scouting hundreds or thousands of players throughout the year were becoming too great. The Combine was a place where NFL scouts could gather to judge player performance at the same time.

Not only did this save time and money, but it also established a universal benchmark. When a team wanted to trade or release a player, other teams could refer to their “benchmark” performance at the combine.

Of course, there are no guarantees in sports. But, essentially, the Combine puts players through a series of drills that are specifically relevant to the sport of football.

However, the Combine is just a small part of the scouting process. In the modern era, teams hold private player workouts so they can determine whether a prospect appears to be a good fit in an organization’s specific system.

Another way of putting all that would be that NFL team developers use a model’s benchmarks as a general performance predictor, but they also conduct rigorous external checks to determine the model’s usefulness in a specific domain.

A player may knock your socks off at the Combine, but if they fail to impress in individual workouts there’s a pretty good chance they won’t make the team.

Ideally, benchmarking in the AI world would simply represent the first round of rigor.

As a team of researchers from UC Berkeley, the University of Washington, and Google Research recently :

Benchmarking, appropriately deployed, is not about winning a contest but more about surveying a landscape— the more we can re-frame, contextualize and appropriately scope these datasets, the more useful they will become as an informative dimension to more impactful algorithmic development and alternative evaluation methods.

News Related

OTHER NEWS

Sign up for Crowd Cow meat deliveries and get $100 off your first 10 boxes—here's how

— Recommendations are independently chosen by Reviewed’s editors. Purchases you make through our links may earn us a commission. There’s plenty of meal kits to choose from, and with current ... Read more »

This epic TikTok sleep hack aims to help you fall asleep in 2 minutes

If you have trouble falling asleep you’re not alone, and right now on TikTok a sleep technique is going viral as a way to help you drop off fast. The ... Read more »

Pufferfish has emergency dental work after teeth grow so big she can't eat

Goldie the porcupine pufferfish was losing weight and was at risk of starving because of her giant teeth. (SWNS) A fish had to be rushed to an animal dentists to ... Read more »

Obsessed with Wordle? Here are 10 similar games to play

— Recommendations are independently chosen by Reviewed’s editors. Purchases you make through our links may earn us a commission. A few weeks ago, several of my friends started posting strange ... Read more »

The 5 best deals you can shop today from Best Buy, Coach Outlet and Sur La Table

— Recommendations are independently chosen by Reviewed’s editors. Purchases you make through our links may earn us a commission. If you’re itching to do some online shopping, we’ve got you ... Read more »

How NASA's space research ends up in earthly products

NASA is best known for exploring space, but the agency’s technology is also touching down on Earth. A new book, called , shines the spotlight on the products that have ... Read more »

Facebook parent Meta creates powerful AI supercomputer

MENLO PARK, Calif. — Facebook’s parent company Meta on Monday said it has created what it believes is among the fastest artificial intelligence supercomputers running today. The social media giant ... Read more »

The best multiplayer PC games 2022: top competitive games to play today

The best multiplayer PC games 2022 has to offer will draw you in, time and time again. That’s how much fun they are. These games will get the adrenaline pumping ... Read more »

HexClad makes our favorite cookware sets, and right now you can get them for 30% off

— Recommendations are independently chosen by Reviewed’s editors. Purchases you make through our links may earn us a commission. When it comes to cookware, we think HexClad is one of ... Read more »

NASA's James Webb Space Telescope successfully positioned to look back in time

The $10 billion James Webb Space Telescope has completed its insertion burn at the second Lagrange (L2) point a million miles, or 1.5 million kilometers, from Earth, successfully completing its ... Read more »

Beyond Webb: why NASA's new $10 billion space telescope is just the beginning

We need more light. Having launched on Christmas Day, the James Webb Space Telescope is now at its destination and preparing to make some incredible observations, but the race to ... Read more »

New rumors reveal Intel Core i5-12500H and i7-12700H beats out Ryzen 7 5800H in benchmark tests

According to newly leaked benchmarks, the mid-range 12th-generation Alder Lake mobile processors score surprisingly high, just in time for some new gaming laptops to hit the market. The test leaks ... Read more »

New space telescope reaches final stop million miles out

The Associated PressFILE – This 2015 artist’s rendering provided by Northrop Grumman via NASA shows the James Webb Space Telescope. On Monday, Jan. 24, 2022, the world’s biggest and most ... Read more »

Dominic Cummings refuses to meet Sue Gray so PM can't ‘spin his way out'

Dominic Cummings refuses to meet Sue Gray (PA Archive) Dominic Cummings said he won’t meet with Sue Gray so Boris Johnson can’t “invent nonsense and spin it in the media”. ... Read more »

Peter Dinklage blasts Disney for remaking ‘f***ing backwards' Snow White and the Seven Dwarfs

Peter Dinklage has criticised the forthcoming revival of the 1937 children’s film Snow White and the Seven Dwarfs, calling the story “f***ing backwards”. The actor, who has dwarfism, was discussing ... Read more »

Meta says ‘fastest' AI supercomputer will detect harmful content

The company, helmed by CEO Mark Zuckerberg, says its artificial intelligence supercomputer will debut later this year. NurPhoto via Getty Images Facebook parent Meta said on Monday that its research ... Read more »

Chimpanzee culture ‘more similar' to humans' than previously realised, researchers say

The nut cracker. A chimpanzee in Guinea uses a hammer and anvil arrangement of stones to crack open a nut (Kathelijne Koops, UZH) Human culture is not only complex, but ... Read more »

Dark Souls 3 bug could let hackers seize control of your PC

As if Dark Souls 3 wasn’t already difficult enough, playing it online could open up your computer for malicious actors to swoop in, steal sensitive data, and brick it completely, ... Read more »

Best single-player games 2022: the top games for playing solo on console and PC

Dive into the best single player games available right now for a gaming experience you’ll never forget. These are titles big and small that have managed to not only capture ... Read more »

How to watch The Responder online for free and from anywhere

Night after night, Chris Carson navigates the dark streets of Liverpool responding to emergency calls, but worn down by the stresses of the job and issues at home, he decides ... Read more »

IIT-Guwahati team develops 3D printing tech to cut concrete use by 75%

A team of researchers at the Indian Institute of Technology Guwahati (IIT-G) has developed a 3D printed urban furniture using construction material made from local industrial wastes.Concrete 3D printing is ... Read more »

Publishers ask EU to stop Google from removing cookies from Chrome

Hundreds of European media companies and advertisers want to stop Google from phasing out its third-party cookies. NurPhoto via Getty Images German media companies want the European Union to prevent ... Read more »

3 Questions: Anuradha Annaswamy on building smart infrastructures

Senior research scientist and her team are designing intelligent systems that could someday transform the way we travel and consume energy. Read more »

Astonishment as SNP claims ‘clear signs of Covid recovery' pave way for second independence vote

John Swinney insisted that some sectors are ‘in a stronger position today’ than before the pandemic – Fraser Bremner/Daily Mail/PA Wire Hospitality leaders have expressed astonishment after Nicola Sturgeon's deputy ... Read more »

Meat Loaf's home with wife Deborah didn't pay tribute to music career

After Meat Loaf, whose real name is Michael Lee Aday, sadly passed away at the age of 74, tributes have been pouring in from his followers remembering his incredible music ... Read more »

Meghan King Goes on 'Empowering' Solo Ski Trip to Utah After Cuffe Owens Split: 'Here's to Me'

Meghan King Goes on ‘Empowering’ Solo Ski trip to Utah After Cuffe Biden Owens Split Meghan King/Instagram Meghan King is traveling solo! Last week, the Real Housewives of Orange County ... Read more »

Sarah Palin tests positive for Covid, delaying New York Times defamation trial

Photograph: Steve Nesius/Reuters ‘She is, of course, unvaccinated’, judge says after positive test Former Alaska governor suing newspaper over 2017 editorial Sarah Palin has tested positive for the coronavirus – ... Read more »

Water seen below polar ice cap on Mars may be optical illusion, scientists say

The extreme conditions on the red planet make the existence of water on its surface unlikely. Scientists now say they have a more plausible explanation, with the sighting in 2018 being put down as a mirage. Read more »

A Minecraft competition inadvertently took a whole country offline - here's how

A multi-day DDoS attack coinciding with a highly-anticipated Minecraft tournament left the entire country of Andorra without access to the internet last week. The DDoS campaign targeted Andorra Telecom, the ... Read more »

Google faces lawsuits over location-tracking practices from U.S. states

Google spokesperson Jose Castaneda said the “attorneys general are bringing a case based on inaccurate claims and outdated assertions about our settings.” Read more »

With Crash Bandicoot and Spyro heading to Microsoft, PlayStation should revive Jak and Daxter

Microsoft’s agreed acquisition of Activision Blizzard for close to $70 billion is a monumental deal that’s thrown up all manner of talking points about the way we play video games, ... Read more »

Ford Unveils The Bronco Raptor, Starting At $70,000

Ford Motor Company (NYSE:F) has announced a new Bronco SUV member, the Raptor, to be released later this year and starting at $70,000. The automotive giant is also aiming at ... Read more »

Best free DVD burner 2021: burn videos and data to disc

The best free DVD burners make it easy to burn videos and data to disc. Best free DVD burner 1. Ashampoo Burning Studio Free 2. WinX DVD Author 3. BurnAware ... Read more »

These Amplifiers Upgrade Your Home Wi-Fi

The business world has adapted to a new normal over the past couple of years as businesses have shifted to fully remote or hybrid models. If your company expects you ... Read more »

GT7 could be the PS5 game that forces your SSD expansion

Gran Turismo 7’s file size has been revealed, confirming that the upcoming racing game will eat your console’s storage space much like the cars it features would eat your wallet. ... Read more »

When will the Motorola Edge 30 be released in the UK? Here's everything you need to know

The handset is expected to arrive in the summer (iStock/The Independent) Motorola might not be the dominant force it once was – we still remember our Razr flip phone fondly ... Read more »

Bose SoundLink Flex review

One-minute review For a small, relatively affordable product, the portable Bluetooth speaker has an awful lot of boxes to tick before it can be deemed ‘competitive’. Happily, the Bose SoundLink ... Read more »

Why Implementing IT Service Management Is More Crucial Than Ever

After suffering from the catastrophic pandemic in 2020 and 2021, this is an important year for organizations operating in major industries worldwide. The Covid-19 situation resulted in a sharp drop in the demand ... Read more »

This Outlook email update will give your calendar a splash of color

Microsoft Outlook users will soon be able to liven up their work calendars with a bit of color thanks to a new update coming to the service. An upcoming upgrade ... Read more »

Best registry cleaners of 2022: Free and paid programs for Windows 10

Best registry cleaners 1. Restoro 2. CCleaner 3. Iolo System Mechanic 4. Wise Registry Cleaner 5. AVG TuneUp Read on for our detailed analysis of each program In the modern ... Read more »
On free-english-test.com you will find lots of free English exam practice materials to help you improve your English skills: grammar, listening, reading, writing, ielts, toeic