What is the Ceiling for Nvidia backed Japan based Sakana AI (さかな)?

Michael Kevin Spencer

Sep 17, 2024 — 9 min read

Hey Everyone,

I've been mulling over the announcement of Sakana Labs, an offshoot of the famous LLM paper who have their startup based in Japan.

Sakana AI, based in Tokyo, Japan, is taking inspiration from nature to build the next generation of AI and it's an intriguing approach to the future of Generative AI. I'm fairly sure David Ha is a genius.

However the AI Scientist PR was a blatant publicity stunt and it's hard to tell how serious this AI startup actually is. The things you have to do to get noticed by VCs and the world today. Well it worked.

On September 4th, 2024 Sakana Labs the Japanese startup founded by former Google researchers David Ha and Llion Jones and former diplomat Ren Ito, has announced it has raised $100 million in a series A round of funding. That's a rather large amount again!

The funding round, led by New Enterprise Associates, Khosla Ventures, and Lux Capital, also saw participation from NVIDIA, with whom Sakana AI has formed a strategic collaboration.

They call themselves an R&D company on a quest to create a new kind of foundational AI model based on nature-inspired intelligence. On August 13th, 2024 they introduced The AI Scientist, the first comprehensive system for fully automatic scientific discovery, enabling Foundation Models such as Large Language Models (LLMs) to perform research independently.

In collaboration with the Foerster Lab for AI Research at the University of Oxford and Jeff Clune and Cong Lu at the University of British Columbia, they were excited to release their new paper, The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.

In it they reported on:

A run a fully AI-driven system for automated scientific discovery, applied to machine learning research.
The AI Scientist automates the entire research lifecycle, from generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript.
They introduced an automated peer review process to evaluate generated papers, write feedback, and further improve results. It is capable of evaluating generated papers with near-human accuracy.
The automated scientific discovery process is repeated to iteratively develop ideas in an open-ended fashion and add them to a growing archive of knowledge, thus imitating the human scientific community.
In this first demonstration, The AI Scientist conducts research in diverse subfields within machine learning research, discovering novel contributions in popular areas, such as diffusion models, transformers, and grokking.

It's hard to know if this can become a real product or not. If Khosla Ventures and Nvidia thinks so, maybe they really are on to something? It's the sort of thing big-wig Eric Schmidt often talks about publically on podcasts and so forth. A lot of AI CEOs are using the science "AI for good" card to market their products and seem like benevolent actors in the world. It's a PR strategy that Google and Microsoft push fairly hard.

A lot of the authors of the famous paper at Google that started the age of Generative AI have gone on to found companies, have exits and basically get very wealthy. Jones was one of the authors of the seminal 2017 Google research paper "Attention Is All You Need", that introduced an AI architecture called transformers.

Japan is actually really interesting territory for the future of technology with many startups having hubs there. David Ha's team has done well getting local partners as well in recent months especially:

Sakana AI as a corporate name sakana is derived from the Japanese word さかな (sa-ka-na) which means fish. Their logo is meant to invoke the idea of a school of fish coming together and forming a coherent entity from simple rules as we want to make use of ideas from nature such as evolution and collective intelligence in our research. The red fish swimming away represents a desire to not simply do what everyone else does, but to pursue what they believe is coming next!

Saying that your AI can do research is very much what everyone else is doing, by the way. But the entire concept and geography of the company is unique but their sizeable funding round not so much. Sakana AI aims to build “a world-class AI lab” in Japan to help the country and its allies meet the challenges they are facing, including a declining population, decreased competitiveness and growing geopolitical tensions, according to the post.

They have managed to garner the support of the broad spectrum of Japanese enterprises as our strategic investors, including the 3 Japanese Megabanks (MUFG, SMBC, Mizuho) and other leading Japanese companies in each of 🇯🇵 industry sectors. What does an AI startup have to do with finance or these national challenges I do not know.

David Ha who is the CEO, was previously head of Google's AI research team in Japan and most recently led research at AI startup Stability AI before they imploded with rather poor leadership. Together, Jones and Ha aim to develop new kinds of AI models based on principles they've gleaned from observing natural systems.

Read Series A announcement

Sakana AI is as of mid September, 2024 now one of the fastest growing startups in Japan. It's fetched a valuation of $1.5 billion in just over a year since founding. It's backed by Nvidia and Japan's three megabanks including MUFG among others.

Image: interview with Bloomberg.

So for Japan and this Sovereign AI movement such a startup could actually be a big deal. It's just a 20 person startup so far. According to a recent Bloomberg piece Sakana’s focus on a large number of small models that work together, rather than on one large foundational model, is attracting investor interest in resource-poor Japan. The island nation, which relies almost wholly on imported energy, has been exploring more power-efficient ways to catch up in AI.

David Ha of course has some background in how Japanese culture works. Ha previously managed Goldman Sachs’ Japanese fixed-income trading business, and that “provided me with an incredible education of how the Japanese economic system actually worked,” Ha said. “I’m excited to bring my expertise from various parts of my career to build the best team that can help accelerate Japan with AI technology.”

It's difficult to know how late to the game or how novel solutions Sakana Labs can bring to the table. Sovereign AI has not gone too well in 2024, with many promising startups seeming to be unable to compete with the pace of scale and GPU compute required. Only makers in China really seem to be able to match the pace on a shoestring budget. I doubt very much Japan is at the same level of hustle and talent.

That being said, I am expecting and actually hoping for some unlikely winners too, and not just the American Cloud funded monopoly picking the winners sort of capitalism here in the future of AI.

AI Scientist Paper

Why did Nvidia and other partners invest in Sakana AI?

“Countries are embracing Sovereign AI to capture and codify their data, culture and language through their own unique large language models. The team at Sakana AI is helping spur the democratization of AI in Japan by developing cutting-edge foundation models to automate and speed up scientific discovery with NVIDIA’s accelerated computing platform.”

—Jensen Huang, Founder and CEO of NVIDIA

“While many labs globally are trying to catch up, and train foundation models by using the same techniques as everyone else, Japan’s Sakana AI is showing the path to innovation. We have been early investors and are thrilled to have NVIDIA join us on this journey.”

—Vinod Khosla, Founder of Khosla Ventures

Sakana AI’s collaboration with Nvidia will focus on research, infrastructure and AI community building in Japan, according to the post. Sakana AI is making itself embedded within Japanese priorities and thinking. It's a fairly good survival strategy when LLM building and talent acquisition is so expensive in a Generative AI bull market.

Jones reportedly said at the time that he didn’t want Sakana AI to be “just another company with an LLM.” So it's time for Sakana Labs with all of this cash to actually prove it. $100 million is a lot of money for a startup with a dream and not even dozens of employees.

For more information on the AI Scientists concept:

An Overview of how The AI Scientist works.
More Examples of generated papers and innovations discovered by The AI Scientist.
Known Limitations and Challenges faced by the current version of The AI Scientist.
Interesting and unexpected things The AI Scientist sometimes does in order to increase its chance of success, such as modifying and launching its own execution script! We discuss the AI safety implications in our paper.
A Discussion about ethical and broader future implications of The AI Scientist.

For more details and many more example papers, please see our full scientific report. We are also releasing open source code and full experimental results on our GitHub repository.

Jones and Ha intentionally went against the grain, believing Tokyo offers key advantages, but are they actual innovator or gaming the macro environment and geopolitical risks by camping out in the far east?

The AI Scientist scheme doesn't sound legit for 2024. The so-called "innovative system" uses large language models (LLMs) to mimic the scientific process. It can according to the media, generate research ideas, design and execute experiments, analyze results, and even perform peer reviews of its own papers. The researchers claim that The AI Scientist can produce a complete research paper for approximately $15 in computing costs.

Considering the explosion of AI papers and academic papers in general often with ChatGPT elements, Generative AI's impact on science may not be as good as is often portrayed. Deskilling academics doing the hard work of research analysis could be problematic. A PhD candidate is expected to be performing original, independent research with the help of a supervisor and a co-supervisor. Not is all what it seems.

A lot of claims are being made in how good Generative AI is at coding and science but we need to be very careful. Sakana AI employed a technique called "model merging" which combines existing AI models to yield a new model, combining it with an approach inspired by evolution, leading to the creation of hundreds of model generations.

An AI startup needs to do certain things in order to be successful. The idea of AI paper writers isn't the best forms of automation that are useful in the real world I can think of. Real science requires rigor, and the grounding of an actual lab and real professors who are experts in their field. Showing that Generative AI can mimic the output is ah, a bit deceptive.

But hey if it helps you raise $100 million dollars who is anybody to complain, most won't care that you accomplished such a feat in such a fishy way. Most of these high flying Generative AI startups won't make the grade, they will have to swim upstream with too much dilution.

This massive round comes just six months after the company's $30 million seed round, bringing its total funding to more than $130 million within its first year of operation. That's a lot for a Japanese AI startups. Most of the talent will of course be American. This school of fish are digital nomads in the most extreme sense, not native to Japan.

Vinod Khosla, founder of Khosla Ventures, praised Sakana AI's innovative approach: "While many labs globally are trying to catch up and train foundation models using the same techniques as everyone else, Japan's Sakana AI is showing the path to innovation." But who knows if that is correct?

In fact, I cannot think of a major Japanese AI startup. Sakana Labs is a transplant using Tokyo as a hub.

Do you suppose Llion Jones has good Japanese at this point? This is just a Google-brain spinoff pretending to be something else. Welcome to the world of wacky hallucinations where Generative AI startups even make up an entire narrative around how they are different or unique. Until proven otherwise, I'll just assume these are marketing gimmicks and ideological constructs that will be readily abandoned for commercial priorities as these things usually go.

Let's build real products and we'll see about the ceiling of this startup. It says it will use the capital as well as infrastructure support from Nvidia to further advance its technologies and grow into a world-class AI research lab taking on frontier labs, including those of OpenAI and Anthropic. I see no evidence thus far that it's capable of doing so. Nor do I believe "Sovereign AI" in an era of American exceptionalism is off to a good start. Nvidia has made some good picks investing in Generative AI companies, so I suppose it has its reasons.