WEKA

“Our storage system went from complete chaos to order.”

“NeuralMesh made it possible to train models at the scale we needed.”

“Within just half a year, we trained a foundation model from scratch that is competitive by global standards,”

“We have not had a single training fail due to the storage solution since we implemented WEKA,”

“There were occasions where WEKA came to us and said: we are seeing something weird in your cluster, are you sure it’s OK,” Lee says. “That is quite rare when working with a software company.”

“WEKA was the only one that satisfied all our requirements.”

““With WEKA, I know that the data I have is accessible where it needs to be. We don’t have to worry about storage,” said Dr. Young.”

““The previous solution was at 50-100 TB scale, whereas now we’re at much more.””

““We now provide persistent areas for some user datasets, like the AlphaFold databases, so people can use that rather than the scratch storage where they have to copy data up,” Dr. Young said. “Otherwise they waste ten minutes of a day-long computational run just copying data up and pulling results back down.””

““We have access to support through a Slack channel, which has been great,” Dr. Young said.”

““WEKA came to the table and helped solve the problem for us,” Dr. Young said. “We were very glad that the presales team stepped up and made our hardware work for us.””

““Our storage needs to be able to work across both InfiniBand and Ethernet fabrics, which WEKA does easily for us,” Dr. Young said. “It’s able to present the system we have on NeuralMesh to both fabrics without us having to worry about it.””

““With NeuralMesh, we know that one client can’t kill the whole storage system.””

““WEKA won the competition we ran to find out what high-performance storage we should go for. Part of the criteria was, can it feed the data to our compute and GPU resources appropriately running on the networking we have now and the networking we might have in the next two to five years.””

““We knew we wanted to get faster storage, able to cope with more compute resources calling on it. That probably was a motivator, but not necessarily a wall.””

““We were worried about a rogue job causing a problem to the storage , which caused more than just a problem for that job, but a problem for everyone across the whole of our estate.””

““Before WEKA, our main concern was that a single component failure could propagate beyond the affected subsystem and compromise adjacent parts of the environment.””

““Whenever we needed to fix things or make things work, or things broke, there were headaches. Moving to upgrade or fixing things by installing new firmware involved varying amounts of pain,” Dr. Young said. “We were looking for a solution that didn’t introduce more pain points and didn’t break things in different ways. Serviceability is maybe the best word for it. We wanted to get away from the pain of not being able to keep current with the manufacturer’s recommendations.””

““We get to play with the new toys, but what really matters is that we provide the expertise that enables researchers to access hardware they wouldn’t be able to procure within their own research group. We provide the scale for people to do larger, more significant work on this central resource. What gets me excited is that we’re helping researchers across the university achieve their research goals.””

“What I love about WEKA is the engineer engagement. They’re very much a technical entity, just like ourselves. We’re out to solve problems and bring value to customers.”

“With this capability and working with the WEKA team, we can engineer out obsolescence . We can take an existing GPU-based system and bring it to market ready for the next generation of workloads, without having to redeploy, throw out the old, and bring in a whole new system.”

“Highly performant data infrastructure is essential to ensuring our customers’ demanding AI workloads operate at peak efficiency. Within our Sustainable AI Factories, a steady, frictionless data pipeline is also essential to further ensure ongoing energy savings. Because of this, WEKA’s NeuralMesh solution was a natural choice. Their high-performance platform, combined with our energy-efficient GPU clusters, embodies our mutual commitment to not just innovate with AI but to do so responsibly.”

“Now, when a researcher worries that with the base compute allocation it will take six months or a year to chew through their parameter sweep, we can say, ‘if you have the budget, we have the tools to make it happen.’ That’s the power NeuralMesh gives us.”

“They’re just wowed at how much better and more consistently their jobs are running,”

“With NeuralMesh, I no longer require a full-time employee dedicated to file system hardware administration. The ability to manage this cluster with reduced overhead is a significant relief.”