IBM commits high-scale inference platform ModelMesh to open source

artificial intelligence, Open source, IBM, DevOps, Artificial intelligence (video games), Watson, Neural

IBM today announced it has committed its inference service to open source. This is a big deal for the MLOps and DevOps community, but the implications for the average end-user are also huge.

Artificial intelligence is a backbone technology that nearly all enterprises rely on. The majority of our coverage here on Neural tends to discuss the challenges involved in training and developing AI models.

But when it comes to deploying AI models so that they can do what they’re supposed to do when they’re supposed to do it, the sheer scale of the problem is astronomical.

Think about it: you log in to your banking account and there’s a discrepancy. You tap the “How can we help?” icon at the bottom of your screen and a chat window opens up.

You enter a query such as “Why isn’t my balance reflecting my most recent transactions?” A chat bot responds with “One moment, I’ll check your account,” and then, like magic, it says “I’ve found the problem” and gives you a detailed response concerning what’s happened.

What you’ve done is sent an inference request to a machine learning model. That model, using a technique called natural language processing (NLP), parses the text in your query and then sifts through all of its training data to determine how best it should respond.

If it does what it’s supposed to in a timely and accurate manner, you’ll probably walk away from the experience with a positive view on the system.

But what if it stalls or doesn’t load the inferences? You end up wasting your time with a chat bot and still need your problem solved.

ModelMesh can help.

Animesh Singh, IBM CTO for Watson AI & ML Open Tech, told Neural:

ModelMesh underpins most of the Watson cloud services, including Watson Assistant, Watson Natural Language Understanding, and Watson Discovery and has been running for several years.

IBM is now contributing the inference platform to the open source community.

Designed for high-scale, high-density, and frequently-changing model use cases, ModelMesh can help developers scale Kubernetes.

ModelMesh, combined with KServe, will also add Trusted AI metrics like explainability, fairness to models deployed in production.

Going back to our banking customer analogy, we know that we’re not the only user our bank’s AI needs to serve inferences to. There could be millions of users querying a single interface simultaneously. And those millions of queries could require service from thousands of different models.

Figuring out how to load all these models in real-time so that they can perform in a manner that suits your customer’s needs is, perhaps, one of the biggest challenges faced by any company’s IT team.

ModelMesh manages both the loading and unloading of models to memory to optimize functionality and minimize redundant power consumption.

Per an IBM press release:

It is designed for high-scale, high-density, and frequently changing model use cases. ModelMesh intelligently loads and unloads AI models to and from memory to strike an intelligent trade-off between responsiveness to users and their computational footprint.

You can learn more about ModelMesh here on IBM’s .

Internet Explorer Channel Network
News Related

OTHER NEWS

iPhone users, researchers have a 'Facebook warning' for you

Security researchers (via Forbes) have warned iOS users that the Facebook app installed on their device is still harvesting their data with the help of the accelerometer even if they ... Read more »

British scientists ‘frozen out' of EU research due to Brexit row, MPs warn

A delay in the UK’s participation in EU research programmes because of a row about the Brexit deal is beginning to hurt British science, MPs have warned. A report by ... Read more »

Hey, big suspender! Spanx underwear boss gifts all employees first class tickets and $10,000

Watch: Spanx founder surprises workers with two first class tickets to anywhere in the world Spanx employees are currently celebrating as the company's founder has recently gifted staff two first ... Read more »

Macy's early Black Friday deals are here: Save on Instant Pot, Shark vacuums and more

— Recommendations are independently chosen by Reviewed’s editors. Purchases you make through our links may earn us a commission. Macy’s early holiday sales are finally here. With the kickoff of ... Read more »

Microsoft reinstates Hot Reload tool after huge open-source backlash

Microsoft has caved in to the pressure from the open source community and reversed its decision to yank a key feature from the upcoming .NET 6 release.. Microsoft wrongly rubbed ... Read more »

You can already use Google's Play Store in Windows 11 – here's how

Running Android apps in Windows 11 through Amazon was brought to the Insider program last week, and already a developer has unlocked Google’s Play Store for you to run any ... Read more »

LNG demand to rise 25-50% by 2030: Morgan Stanley

Samsung Heavy Industries’ LNG carrier / Courtesy of Samsung Heavy Industries Demand for liquefied natural gas (LNG) is expected to rise by 25 percent to 50 percent by 2030, making ... Read more »

Lenovo launches Tab K10 with 10.3-inch FHD display, MediaTek Helio P22T and 7500 mAh battery at Rs 25,000

Lenovo has announced the launch of its new tablet Tab K10 In India. The new Android 11-based tablet comes with a 10.3-inch FHD display. The Lenovo Tab K10 is powered ... Read more »

Wu-Tang Clan could lead official soundtrack's production for rumored Xbox game codenamed ‘Shaolin'

Photo credit: Billy Freeman / Unsplash It has been more than a month since a massive list of unannounced games were leaked from the Nvidia GeForce Now network, but fans continue hearing that ... Read more »

One of Microsoft's most-hated products might actually be getting a useful upgrade

Searching for lost files or that extra bit of data to make the difference at work could be quicker than ever thanks to a new upgrade from Microsoft. The tech ... Read more »

Best wireless gaming mouse 2021

There was a time when even the best wireless gaming mouse couldn’t hold a candle to its tethered colleagues, but that’s all in the past. The wireless gaming mice of ... Read more »

Twitter downvote feature may come to web as well

Twitter is reportedly working to expand the downvote feature to more users. According to the known tech blogger, Jane Manchun Wong, the company is working on a downvote button for ... Read more »

macOS Monterey rolling out on October 25: Here are the things you should keep in mind before updating

Announced at Worldwide Developers Conference, macOS Monterey will be rolling out to compatible Mac laptops and desktops on October 25. The operating system brings several new features and changes to ... Read more »

Google Drive for desktop client receives complete support for Apple M1 chip

Google has finally updated its Drive for desktop application with full support for Apple M1 powered Mac laptops and desktops.Earlier this year, Google rolled out M1 support for its previous ... Read more »

Best web development tools in 2021

The best web development tools make it simple and easy for developers to develop and design a custom website platform. Web development has long since left the grounds of having ... Read more »

Best Excel online courses in 2021

The best Excel online courses will make it easy to develop basic or advanced skills using Microsoft Excel by distance learning. Best Excel online courses Click the links below to ... Read more »

Facebook papers: What we know about how misinformation and extremism spread from whistleblower

Misinformation and extremism spreading unchecked. Hate speech sparking conflict and violence in the U.S. and abroad. Human traffickers sharing a platform with baby pictures and engagement announcements. Despite its mission ... Read more »

The Facebook Papers: Documents reveal internal fury and dissent over site's policies

Thousands of leaked documents highlight employees’ disillusionment with spread of misinformation and calls to violence. Read more »

Huawei CFO Meng Wanzhou returns to hero's welcome at company headquarters after 21-day quarantine

Huawei Technologies Co finance chief Meng Wanzhou, who has completed a 21-day Covid-19 quarantine after returning from Canada, came back to her company’s head office to a hero’s welcome on ... Read more »

Electric vehicle savings explored

Driving an electric vehicle can save you a lot of money on gas… but how much money does it really save you? Well, it depends. Read more »

Why Strictly Come Dancing fans are fuming with Anton Du Beke

Photo credit: BBC Strictly Come Dancing fans had two major talking points after Saturday evening's [23rd October] show: Adam Peaty's fairly tight trousers, and Anton Du Beke's mispronunciation of Oti ... Read more »

Oklahoma resumes lethal injections that ‘burn men alive' this week. Seven men may die before they can appeal

Oklahoma may execute seven death row inmates before their legal appeal in federal court can play out. (Images courtesy of Oklahoma Department of Corrections and Justice for Julius; Composite by ... Read more »

Museum bosses back anti-woke charter to guard against rushed and unplanned changes to UK heritage

A statue of Robert Milligan is pictured being removed by workers outside the Museum of London Docklands near Canary Wharf – Reuters Bosses from three of the UK’s leading museums ... Read more »

Megan Thee Stallion's bodyguard snubs F1 interviewer who awkwardly asked her to ‘freestyle rap'

Megan Thee Stallion’s bodyguard snubbed F1 interviewer Martin Brundle after he asked the singer to “freestyle rap”. Footage of the awkward encounter was broadcast on Sky Sports, as the rapper ... Read more »

Facebook whistleblower live: Frances Haugen appears before UK parliamentary committee

The latest on Facebook as data scientist turned whistleblower Frances Haugen prepares to give evidence to MPs on the UK parliament's Joint Committee on the Draft Online Safety Bill from 2.30pm. Read more »

Another popular npm package infected with malware

In an audacious incident, threat actors hijacked the account of the developer of a widely used JavaScript library, UAParser.ja, to replace the legitimate code with malicious one infused with malware ... Read more »

Best Raspberry Pi distros in 2021

Believe it or not, the Raspberry Pi is now eight years old. In this relatively short period the Pi has ushered in a new revolution in computing that stretches far ... Read more »

Elon Musk reveals his crypto buys

Elon Musk said he holds Dogecoin but no Shiba Inu. Photo: AP Tesla (TSLA) CEO Elon Musk, a crypto proponent, revealed he has not invested in meme token Shiba Inu, ... Read more »

'World's most expensive' fish spotted off coast of Britain

‘World’s most expensive’ fish spotted off coast of Britain Read more »

Panasonic unveils prototype battery to help Tesla lower production costs

Panasonic on Monday unveiled a new large prototype battery designed to help Tesla Inc lower electric vehicle (EV) production costs, in a move the Japanese company’s battery chief said would ... Read more »

A world without checkouts: How till-free supermarkets are changing London's grocery game

(ES composite) Last time I walked out of a shop without paying, I had to make a red-faced trip back to Boots. This time — at a Tesco Express by ... Read more »

Electric cars: Price of EVs puts people off switching from petrol and diesel, poll finds

The cost of electric vehicles is the biggest factor putting people off switching from petrol and diesel, exclusive polling for i has revealed. According to the survey, conducted by Redfield ... Read more »

Julian Knight: Online Safety Bill is ‘simplistic and bad politics', Tory MP tells Boris Johnson

Banning anonymous social media accounts is “simplistic and unworkable”, the chair of the Commons Culture, Media and Sport committee has told i. Julian Knight also took aim at Boris Johnson ... Read more »

Nintendo Switch doesn't need better hardware, according to World War Z dev

With the polarizing release of the Nintendo Switch OLED leading many Switch owners to stick with their original models, interest in an eventual upgraded version remains high. A potential Nintendo ... Read more »

Tesla Cybertruck looks unbelievably cool in new off-road footage

Earlier this year, Tesla CEO Elon Musk described his company’s in-development Cybertruck as a “special project” near and dear to his heart – and thanks to some brand new footage ... Read more »

Idea Spectrum Realtime Landscaping Pro 2020 review

Landscape design is usually back breaking work, involving a spade, and most likely a wheelbarrow. But why not forgo all of this and start your work from the comfort of ... Read more »

5 things I wished I knew when I started playing New World

One of the first lessons you learn is to not call this fella “Bob”. (Screenshot: Amazon Games Studio) Aside from some technical issues, Amazon Games Studio's New World, its massively ... Read more »

Five ways to drive business impact from big data

Big data has been a favorite business buzzwords for many years and while the popularity of the term skyrocketed, some were already tired of it by 2014, like Deloitte who ... Read more »

Ethereum's upgrade is finally coming — will it overtake Bitcoin?

The world’s second most valuable cryptocurrency, ether, has been touching all-time highs in price ahead of a major upgrade of its underlying platform, ethereum. Ether is currently worth in aggregate ... Read more »

Best mobile payment apps in 2021

The best mobile payment apps offer a simple way to make and send payments using your smartphone. Mobile apps providing contactless payments have become common since the introduction of Near ... Read more »
On free-english-test.com you will find lots of free English exam practice materials to help you improve your English skills: grammar, listening, reading, writing, ielts, toeic