Adam Marblestone — AI is missing something fundamental about the brain

all these interviews with AI researchers. How does the brain do it, right? Like we're throwing way more data at these LLMs and they still have a small fraction of the total capabilities that a human does. So what's going on? Yeah, I mean this might be the quadrillion dollar question

or something like that. It's it's arguably making argument. This is the most important, you know,

question in science. I don't claim to know the answer. I also don't really think that the answer will necessarily come even from a lot of smart people thinking about it as much as they are. All right, my overall like meta-level take is that we have to empower the field of neuroscience to just make neuroscience a a more powerful field technologically and otherwise to actually be able to crack a question like this. But maybe the the way that we would think about this

now with like modern AI neural nets deep learning is that there's sort of these certain key components of that. There's the architecture, there's maybe hyperparameters of the architecture, how many layers do you have or sort of properties of that architecture? There is the learning algorithm itself,

“how do you train it? You know, back prop, gradient descent, is it something else?”

There is, how is it initialized? Okay, so if we take the learning part of the system, it still may have some initialization of of the weights. And then there are also cost functions. There's like, what is it being trained to do? What's the reward signal? What are the loss functions? Supervision signals? My personal hunch within that framework is that the the field has neglected the role of this very specific loss functions, very specific cost functions.

Machine learning tends to like mathematically simple loss functions, right, predict the next token. You know, cross entropy, these these these these simple kind of computer scientists, loss functions. I think evolution may have built a lot of complexity into the loss functions, actually many different loss functions were different areas turned on at different stages of

development. A lot of Python code basically generating a specific curriculum for what different

parts of the brain need to learn. Because evolution has seen many times what was successful and unsuccessful and evolution could encode the knowledge of the learning curriculum. So in the machine learning framework, maybe we can come back and we can talk about where to the loss functions of the brain come from. Can that can last different loss functions lead to different efficiency of learning? You know, people will say like the cortex has got the universal human learning

algorithm, the special features. What's up? This is a huge question and we don't know. I've seen models where what the cortex you know the cortex has typically this like six layered structure layers and it's like the different sense and layers of a neural net. It's like any one location in the cortex has six physical layers of tissue as you go in layers of the sheet and then those areas then connect to each other and that's more like the layers of a network. I've seen versions of that

where what you're trying to explain is actually just how does it approximate back prop. Yeah. And what is the cost function for that? What is the network being asked you to if you sort of are trying to say it's something like back prop? Is it doing back prop on next token prediction? Is it doing back prop on classifying images or what is it doing? And no one knows.

“But I think I think one one thought about it. One possibility about it is that”

it's just this incredibly general prediction engine. So any one area of cortex is just trying to

predict any basically can it learn to predict any subset of all the variables it sees

from any other subset. So like omnidirectional inference or omnidirectional prediction whereas an LLM is just you see everything in the context window and then it computes a very particular conditional probability which is given all the last thousands of things. What is the very probabilities for all the the next token? Yeah. But it will be weird for a large language model to say you know the quick brown fox blank blank the lazy dog and fill in in the middle versus

do the next token. If it's doing just forward it can learn how to do that stuff in this emergent level of in context learning but natively is just predicting the next token. What if the cortex is just natively made so that any area of cortex can predict any pattern in any subset of inputs given any other missing subset. That is a little bit more like quote unquote probabilistic

“AI. I think a lot of things I'm saying by the way are extremely similar to like what”

Leon McCune would say. Yeah. He's really interested in these energy-based models and something like that is like the joint distribution of all the variables. What is that? What is the likelihood or unlikely hood of just any combination of variables and if I if I clamp some of

sampling for example I can compute okay conditioned on these being set in this state what are and these could be any arbitrary subset of of of variables in the model. Can I predict what any other subset is going to do in sample from any other subset given clamping this subset and I could choose a totally different subset and sample from that subset. So it's omnidirectional inference and so you know that could be there some parts of air of cortex that might be like association

areas of cortex that might predict vision from audition. Yeah. There might be areas that predicts

“things that the more innate part of the brain is going to do because remember this whole thing is”

basically writing on top of the sort of a lizard brain and lizard body if you will and that thing is a

thing that's worth predicting too. So you're not just predicting do I see this or do I see that but is this muscle about to tense am I about to have a reflex where I laugh you know is my heart rate about to go up am I about to activate this instinctive behavior based on my higher low understanding of like I can match uh somebody has told me there's a spider on my back to this lizard part that would activate if I was like literally seeing a spider in front of me and that you you learn to associate

the two so that even just from somebody hearing you say there's a spider on your back yeah let's well let's come back to this and this is partly having to do with with Steve Burns theories which I'm recently obsessed about but yeah but on your podcast with Ilya he said look I'd not aware of any

“any good theory of how evolution encodes high level desires or intentions. I think this is like”

this is like very connected uh to to all of these questions about the loss functions and the cost functions um that the brain would use and it's a really profound question right like like let's say that um I am embarrassed for saying the wrong thing on your podcast because I'm imagining that young lacoon is listening is that's not my theory that you describe energy based models really badly that's going to enact activate in me innate embarrassment and shame and I'm going to want to

go hide and whatever and that's going to activate these innate reflexes um and that's important because I might otherwise get get killed by young lacoons you know like marauding army of of other the french air resources coming for you Adam and so it's important that I have that instinctual response

but of course evolution has never seen young lacoon or known about energy based models or known

“what an important scientist or a podcast is and so somehow the brain has to encode this desire to”

you know not not piss off really important you know people in the tribe or something like this um in a very robust way without knowing in advance all the things that the learning subsystem okay of the brain the part that is learning cortex and other parts uh the cortex is going to learn this world model it's going to include things like young lacoon and podcast and uh evolution has to make sure that that those neurons whatever the young lacoon being upset with me neurons get properly

wired up to the shame response or this part of the reward function um and this is important right because if we're going to be able to seek status in the tribe or learn from knowledgeable people

as you said or things like that exchange knowledge and skills with friends but not with enemies

and we have to learn all this stuff so it has to be able to robustly wire these learned features of the world um learn parts of the world model up to uh these innate reward functions and then actually use that to then learn more right because next time I'm not going to try to piss off young lacoon if he knows me that that I got this wrong um and so uh we're going to do further learning based on that so in constructing the reward function it has to use uh learned information

but how can evolution evolution didn't know but long young lacoon so how can how can it how can it do that and so uh the basic idea um that Steve Burns is proposing is that we'll part of the cortex uh or or other areas like the Immigna law that learn um what they're doing is they're modeling the steering subsystems and the steering subsystems is the part with these more innate innately program responses and the innate programming of these series of reward functions,

cost functions bootstrapping uh functions that exist so they're parts of the Immigna law for example they're able to monitor what what those parts do and predict what those parts do so um so how do you find the neurons um that are important for social status well you have some innate heuristics of social status for example or you have some innate uh innate uh heuristics of friendliness that um that the steering subsystem can use and the steering subsystem actually has its own

sensory system which is kind of crazy so we think of you know vision as being something that the cortex does but there's also a steering subsystem sub-cortical visual system called the superior collculus with innate ability to detect faces for example or threats um so it so there's a visual

so there'll be part of the Immigna law or part of the cortex that is learning to predict those

“responses and so what are the neurons that are that matter in the cortex for um social status or”

for friendship or they're the ones that predicts those innate heuristics for friendship right so you train a predictor in the cortex and you say which neurons are part of the predictor uh those are the ones that are now is now you've actually managed to wire it up yeah this is fascinating um I feel like I still don't understand I understand how the cortex could learn how this primitive part of the brain would respond to um so you can obviously it has these labels on here's literally

a picture of a spider yeah this is bad like we steer to this right and then the cortex learns that this is bad because the innate part tells of that but then it has to generalize to okay the spiders on my back yes and somebody's telling me the spiders on your back that's also bad yes

but it never got supervision on that right so how does it well it's because the learning subsystem

um is a powerful learning algorithm that does have generalization uh that is capable of generalization so the steering subsystem these are the innate responses so you're going to have some let's say built into your steering subsystem uh these lower brain areas hypothalamus brain stem etc and again they include they have their own primitive sensory systems so there may be an innate response um if I see something that's kind of moving fast toward my body that I didn't previously see

was there and it's kind of small and dark and high contrast that might be an insect kind of skittering onto my body um I am going to like flinch right um and so they're these innate responses and so there's going to be some group of neurons let's say in the hypothalamus that is the I am flinching yeah or I just flinched right there the I just flinched at neurons in hypothalamus

so when you flinch first of all that a negative contribution with the reward function you didn't

want that to happen perhaps um but that's only happening that's a reward function then that is it doesn't have any generalization in it so I'm going to avoid that exact situation of the things skittering toward me um and maybe I'm going to avoid some action that lead to the things skittering uh so that's that's something a generalization you can get what Steve calls it is downstream of the reward function um so I'm going to avoid the situation where the spider was skittering toward me

but you're also going to do something else so there's going to be like a part of your mingola say that is saying okay um a few you know a few milliseconds you know hundreds hundreds of milliseconds or seconds earlier um could I have predicted that flinching response it's going to be it's going to be a group of neurons that is essentially a classifier am I about to flinch um and I'm

“going to have classifiers for that for every important steering subsystem variable that evolution”

needs to kick care of am I about to flinch and my talking to a friend should I laugh now is the friend high status whatever variables the hypothalamus brainstem contain um am I about to taste salt um so that's going to have uh all these variables and for each one is going to have a predictor it's going to train that predictor now the predictor that it trains that can have some generalization and the reason it can have some generalizations because it just has a totally different

input so its input data might be things like the word spider right but the word spider can activate and all sorts of situations that lead to the world's word spider activating in your word world model um so you know if you have a complex world model which really complex features that inherently gives you some generalization it's not just the thing skittering toward me it's even the word spider or the concept of spider is going to cause that to trigger and this predictor can

learn that so whatever spider neurons are in my world model um which could even be a book about spiders or somewhere a room where there are spiders or whatever that is um the amount of TV TVs this this conversation is like so now activating your steering system your your steering subsystem spider hypothalamus a subgroup of neurons of of skittering insect are activating based on these very abstract concept in the conversation we're going I'm going to put an

trigger warning that's because that's because you learn this and the the because the cortex inherently has the ability to generalize because it's just predicting based on these very abstract variables and all these integrated information that it has whereas the steering system only can use whatever the superior collect class and a few other sensors yeah spit out so by the way it's remarkable that the person who's made this this connection between different pieces

of neuroscience even burns like for my physicist yeah has for the last few years has been trying to synthesize isn't AI safety researcher he's just synthesizing this comes back to the academic

“incentives right I think that this is it's this is a little bit hard to say what is the exact”

next experiment how I'm going to publish a paper on us how I'm going to train my grass student do is very very speculative but there's a lot in the neuroscience literature and Steven is able to pull this together and I think that Steve has an answer to Ilya's question essentially

which is which is how how does the brain ultimately code for these higher level desires and

link them up to the more primitive rewards yeah very nice question but why can't we achieve this

but remove the masks right the training so it maps every token every token or come up with more labels between video and audio and text so that it's forced to map one teach one I mean that may be that may be the way so it's it's not clear to me some people think that there's sort of a different way that it does probabilistic inference or different learning algorithm that isn't back prop that might be like other ways of learning energy based models or other things like that

that you can imagine but that it's involved in being able to do this and that the brain has that but I think there's a version of it where you know what the brain does is like crappy versions of backprop to learn to predict you know through a few layers and that yeah it's kind of like a multi-modal foundation model right yeah so maybe the cortex is just kind of like a certain kinds of foundation models there you know some LLMs are maybe just predicting the next token but

you know vision models maybe are a trait learning to fill in the blanks or reconstruct different pieces or combinations but but I think that it doesn't an extremely flexible way so it's you know if you train a model to just fill in this blank at the center okay that's great but if you didn't train it to fill in this other blank over to the left then it doesn't know how to do that it's not part of it's like a repertoire of predictions that are like a more ties into the

network whereas with a really powerful inference system you could choose that test time you know

“what is the sub setting of the sub set of variables you need to infer and which ones are clamped”

okay it's two sub questions one it makes you wonder whether the thing that is lacking in our official neural networks is also about the roar function and more about the encoder or the embedding which like maybe the issue is that you're not representing video and audio in text in the right latent abstraction such that they could intermingle and conflict maybe this is also related to why I love to see bad and drawing connections between different ideas like it's like are the

ideas represented at a level of generality right you could would you notice for the problem is these questions are all co-mingles so if you don't know if it's doing a back prop like learning and we don't know if it's doing energy based models and we don't know how these areas are even

connected in the first place it's like very hard to like really get to the ground truth to this

but yeah it's possible I mean I think that people have done some work my friend Joel Depello actually

“did something some years ago where I think he put a model I think it was a model of v1”

of sort of specifically how the early visual cortex represents images and put that as like an input into like a confnet and that like improves some things so it could be it could be like differences the retina is also doing you know motion detection and certain things are kind of getting filtered out so there there may be some preprocessing of the sensory data there may be some clever combinations of which modalities are predicting which or so on that that lead to better representation there

may be much more clever things than that some people certainly do think that there's inductive bias is built in the architecture that will shape the representations you know differently or that there are clever things that you can do so as a sterile which is the the same organization that employs the variants just launched this neuroscience project based on a Doris sauce work and she has some ideas about how you can build

visions systems that basically require less training they put in they it imbilled into the assumptions of the design of the architecture that things like objects are bounded by surfaces and this you know surfaces have certain types of shapes and relationships of how they acclude each other and stuff like that so you you may be possible to build more assumptions into the network

“and evolution may have also put some changes of architecture um it's just I think that also the”

cost functions and so on maybe a key a key thing that it does so any Jones is this amazing

2021 paper where he uses office euro to show that you can trade off test time compute and training compute and well that might seem obvious now this was three years before people were talking about inference scaling so this got me thinking is there an experiment you could run today even if it's a toy experiment which would help you anticipate the next scaling paradigm one idea had was to see if there was anything to multi agent scaling basically if you have a fixed budget of training

compute are you gonna get a smartest agent by dumping all of it into training one single agent or by splitting that compute up amongst a bunch of models resulting in a diversity of strategies that get to play off each other I didn't know how to turn this question into a concrete experiment though so I started brainstorming with Gemini three pro in the Gemini I am Gemini helped me think through a bunch of different judgment calls for example how do you turn the training loop from

self play to this kind of co-evolutionary lead training how do you initialize and then maintain diversity amongst different office euro agents how do you even split up the compute between these

forked and opened up in anti-gravity which is Google's Asian first IDE the code was originally written

“in 2017 and it was meant to be trained on a single GPU of that time but I needed to train multiple”

whole separate populations of office euro agents so I needed to speed things up I rented a beef cake of a GPU node but I needed to refactor the whole implementation to take advantage of all this scale and parallelism Gemini is just a two different ways to parallelize self-lair one which would involve higher GPU context switching and the other would involve higher communication overhead I wasn't sure which one to pick so I just asked Gemini and not only did it get both of them

working in minutes but it autonomously created and then ran a benchmark to see which one was best it would take me a week to implement either one of these options think up at how many judgment

calls a software engineer working on an actually complex project test to make if they have to

spend weeks architecting some optimization or feature before they can see whether it will work out they will just get to test out so many fewer ideas anyways with all the self from Gemini

“I actually ran the experiment and got some results now please keep in mind that I'm running”

this experiment on an anemic budget of compute and it's very possible I made some mistakes in implementation but it looks like there can be gains from splitting up a fixed budget of trading compute amongst multiple agents rather than just dumping it all into one just to reiterate how surprising this is the best agent in the population of 16 is getting one 16th the amount of trading compute as the agent trained on self-play alone and yet it still outperforms the agent that is

hogging all of the compute the whole process of vibocoding this experiment with Gemini was really absorbing and fun given me the chance to actually understand how helpful zero works and to understand the design space around decisions about the hyper parameters and how search is done and how you do this kind of co-evolutionary training rather than getting bogged out in my very novice abilities as an engineer go to Gemini.google.com to try it out

I want to talk about the study that you just glance off of which was amortized in friends and maybe I should try to explain what I think it means because I think it's probably wrong and this will help you correct for you. Right now the way the model's work is you have an input it maps it to an output and this is amortizing a process that the real process which we think is like what intelligence is which is like you have some prior over how the world could be like

what are the causes that make the world the way it is and then when you see some observation

“you should be like okay here's all the ways the world could be this cause explains what's happening”

best now like doing this calculation over every possible cause is computationally intractable so then you just have to sample like oh here's a potential cause does this explain this observation no forget it let's keep sampling and then eventually get the cause the cause then the cause explains the observation and then this becomes your posterior that's actually pretty good I think of sort of yeah yeah this is Bayesian inference like in generals like of this very

intractable thing right it the algorithms that we have for doing that tend to require up taking a lot of samples multi-carlo methods taking a lot of samples yeah and taking samples takes time I mean this is like the original like Baltimore machines and stuff we're using yeah techniques like this um and still it's used with probabilistic programming other types of methods often

and so yeah so the Bayesian inference problem which is like basically the problem of like perception

like given some model of the world and given some data like how should I update my what what what what are the likes of the variables you know missing variables in my internal model and I guess you idea is that neural networks are hopefully obviously there's mechanistically the neural network is not starting with like here is my model of the world and I'm going to try to explain this data but the hope is that instead of starting with um hey does this cause explain the

saturation node here did this cause explain this explanation yes what you do is just like observation what's the what's the cause that we the neural net things is is the Bayesian observation two cause so the feet forward like goes observation two cause observation two cause to the not the unknown yes you don't have to you don't have to evaluate all these energy values or whatever and and and sample around to make them higher and lower um you just say um approximately that

process would result in this being the top one or something like that yeah one way to think about it might be that test time compute inference time compute is actually doing this sampling again because you literally read its sheet of thought it's like actually doing this toy example we're talking my words like oh can I just all this while I'm by doing X yeah I'm gonna need a different approach

which required inference time compute to elicit get this still into the model so you're

“amortizing the thing which previously needed to do these like rollouts right my declarer rollouts to”

to figure out and so in general there maybe there's a principle of digital minds which can be copied have different trade-offs which are relevant than biological minds which cannot and so in general it should make sense to amortize more things because you can literally copy the copy the amortization right or copy the things that you have um sort of like built in yeah um and it this is a tangential question where it might be interesting to speculate about

in the future as these things can become more intelligent and the way we train them becomes more economically rational what will make sense to amortize into these minds which evolution did not think it was worth amortizing into biological minds I do that you have to

retrain it right I mean first of all I think the probabilistic AI people would be like of

course you need test time compute because this inference problem is really hard and the only ways we know how to do it involve lots of test time compute otherwise it's just a crappy approximation

“that's never gonna like you have to infinite data or something to like make this so I think”

some of the probabilistic people would be like no it's like inherently probabilistic and like amortizing it in this way like just doesn't make sense and so and they might then also point to the brain and say okay well the brain the neurons are kind of stochastic and they're sampling and they're doing doing things and so maybe the brain actually is doing more like the non-amortized inference that real inference but it's also kind of strange how perception can work in just like

milliseconds or whatever it doesn't seem like it uses that much sampling so it's also clearly also doing some kind of baking things into like approximate forward passes or something like that to do that and yeah so in the future you know I don't know I mean I think is it already a trend to some degree that things that are people are having to use test time compute for are getting like used to train back the the base model right yeah that's what now we can do it in one pass right

yeah so I mean I think yeah you know maybe evolution did or didn't do that I think evolution still has to pass everything through the genome right to build the network so and the environment in which humans are living is very dynamic right and so maybe that's if we believe this is true that that there's a learning subsystem per Steve Burns and a steering subsystem that the learning subsystem doesn't have a lot of like pre-initialization or pre-training as is an architecture

but then within life time it learns then evolution didn't you know actually like it immortalized that much into that network it immortalized it instead into instead of an eight behavior is in a set of these bootstrapping cost functions or ways of building up very particular rewards signals yeah this framework helps explain this mystery that people have pointed

“out and I've asked if you're guess about which is if you want to analyze evolution to pre-training”

well how do you explain the fact that so little information is conveyed through the genome so three gigabytes is a size of the total human genome obviously small fraction of that is actually relevant to coding at the brain yeah and if previously people made this analogy that actually evolution has found the hyper parameters of the model the the numbers which tell you

how many layers should there be the architecture basically right like it what how how should things

be wired together but if a big part of the story that increase the sample history aids learning generally makes systems more performant is the reward function is the loss function yeah and if evolution found those loss functions which aid learning then it actually kind of makes sense how so you can like build an intelligence with so little information because like the reward function and you're like right in python right the reward function was a literally a line

yeah and so you just like have like a thousand lines like this and that doesn't take out that much space yes and it also gets to do this generalization thing with the thing the thing I was describing where we were talking with about the spider right of where it learns that just the word spider you know triggers the spider you know reflex or whatever um it gets to exploit that too right so it gets to build a reward function that actually has a bunch of generalization in it

just by specifying these innate spider stuff and the thought assessors as Steve calls them that do the learning um so that's like potentially a really compact solution um to building up these more complex reward functions too that you need so it doesn't have to anticipate everything about the future of the reward function just has to anticipate what variables are relevant what are heuristics for like finding what those variables are um and then yeah so then it has to have like a

very compact specification for like the learning algorithm and basic architecture of the learning subsystem and then it has to specify all this python code of like all this stuff about the spiders and all this stuff about friends and all this stuff about your mother and all this stuff about meeting and um and in social groups and joint eye contact it has to specify all that stuff and so is this really true and so I think that there is some evidence for it so so so so

single cell outlices so one of the things that um neuroscience technology or some scaling up neuroscience

“technology again this is kind of like one of my obsessions um has done uh through through the”

brain initiative big you know neuroscience funding program is they've basically gone through different

areas especially the mouse brand and map like where are the different cell types um how many different types of cells are there in different areas of cortex are they the same across different areas and then you then you look at these sub-cortical regions which are more like the like steering subsystem or reward function generating regions how many different types of cells do they have and which neurons types do they have we don't know how they're all connected and exactly

what they do or what the circuits are what they mean but you can just like quantify like how many different kinds of cells are there um with sequencing the RNA and there are a lot more weird and diverse in bespoke cell types in the steering subsystem basically then there are on the

learning subsystem like the cortical cell types there's enough to build it seems like there's enough

“to build a learning algorithm up there in specifies of hyper parameters and in the in this steering”

subsystem there's like a gazillion you know thousands of really weird cells which might be like the one for the spider flinch reflex and the one for I'm about to taste salt and so why would um each reward function need a different cell type well so this is where you get a nearly wired circuits right so in the in the learning algorithm part in this in the learning learning subsystem you set that's why the initial architecture is best way of learning algorithm is all all the all

the all the juices is happening through plasticity of the synapses changes of the synapses within that big network but it's kind of like a relatively repeating architecture um how is initialized

it's just like um the amount of Python code needed to make you know a eight layer transformer

is not that different from one to make it a three layer transformer right you're just replicating yeah whereas all this Python code for the reward function you know if superior clicklist sees something that's skittering and it leaves you know you're feeling goose bumps on your scan or whatever then trigger spider reflex that's just a bunch of like bespoke species specific situations specific crap that note the cortex doesn't know about spiders it just knows about layers

and but we're seeing the the only way to have this like write this reward function yeah

“used to have a special cell type yeah yeah well I think so I think you have to have a special cell”

types or you have to somehow somehow otherwise get special wiring rules that evolution can say this neuron needs to wire to this neuron without any learning and the way that that is most likely to happen I think is that those cells express like different receptors and proteins that say okay when this one comes in contact with this one let's form a synapse this one so it's genetic wiring yeah and those needs cell types to do it yeah I'm sure this would make a lot more sense of

I knew one of one neuroscience but like it seems like there's still a lot of complexity or generality rather in the steering system so in the steering system so in the steering system has this own visual system that's separate from the visual cortex different features still need to plug into that vision system and the so like the spider thing needs to plug into it and also the the uh love thing needs to plug into it etc etc yes so it seems complicated like I know

it's still complicated and that's that's all the more reason why a lot of the genomic you know real estate in the genome and in terms of these different cell types and so on would go into wiring up the steering subsystem and you can meet cell pre wiring it can me tell how much of the genome is like clearly working so I guess you could tell how many are relevant to the producing the RNA the the manifest or the originators that manifest in different

cell types in the brain right yeah this is what the cell types helps you get at it and I don't think I don't think it's exactly like oh this percent of the genome is doing this but you could say okay and these all these steering subsystem subsypes you know how many different genes are involved in sort of specifying which is which and how they wire um and how much genomic real estate do those genes take up versus the ones that specify you know visual cortex versus audio auditory cortex you kind of just

reusing the same genes to the same thing twice whereas the spider reflex sucking up yes you're right they have to they have to build the vision system they have to build some auditory systems and touch systems and navigation type systems so you know even feeding into the hippocampus and stuff like that there's head direction cells even the fly brain it has innate circuits that you know figure out its orientation and help it navigate in the world and it uses vision figure as optical flow of how it's

flying and you know how is it how is it's flight related to the wind direction it has all these innate stuff that I think in the mammal brain we would all put that and lump that into the steering

fly has to do we're going to have stuff like that too just all in the steering subsystem but do we

do we have some estimate of like here's how many nucleotides here are many megabases it takes to

“I don't know I mean but but but I mean I think you might be able to talk to biologists about this”

you know to to some degree because you can say well we just have a ton in common I mean we have a lot in common with yeast from a genes perspective used to still use as a model for you know some amount of drug development and stuff like that in biology and so so much of the genome is just going towards you have a cell at all it can recycle waste it can get energy it can replicate and then then you see why we have in common with a mouse and so we do know at some level

that the you know the difference is us in a chimpanzee or something and that includes the social instincts and the more advanced you know differences in cortex and so on it's it's a it's a tiny number of genes that go into these additional amount of making the eight layer transformer instead of the six layer transformer are weaking that reward function this would help explain why the hominid brain exploded in size so fast which is presumably like tell me this is correct but under the story

social learning or some other thing increased the ability to learn from the environment and like increased or sample efficiency right instead of having to go and kill the bore yourself and figure out like how to do that you can just be like the elder told me this how you make a spear and then now it increases the incentive to have a bigger cortex which is going to like learn these things yes and that can be done with a relatively few genes because it's really it's really replicating

what the mouse already has is making more of it it's maybe not exactly the same and there may be tweaks but it's like from a perspective you don't have to reinvent all this stuff right so then how far back in the history of the evolution of the brain

does the cortex go back it is the idea that like the cortex is always I have figured out this

omnidirectional inference thing that that's been it's all for a long time and then with the big unlock with primase is this we got the reward function which increased the returns to having omnidirectional inference or is this discussion is the cortex is the omnidirectional inference also something that took a while to tell me I'm not sure that there's agreement about that I think there might be specific questions about language you know are there tweaks to be you know whether that's to

auditory and memory some combination auditory memory region there may also be like macro

“wiring right of like you need to wire auditory regions into memory regions or something like that”

and into some of these social instincts to get I see language for example that so there might be but that might be also a small number of gene changes yeah to be able to say oh I just need from my temporal lobe over here going over to the auditory cortex something right and there is some evidence for the you know the brockers area where Nicky's area they're connected with these hippocampus and so on and so prefrontal cortex so there's like some small number of genes

maybe for like enabling humans to really properly do language that could be a big one but yeah I mean I think that is it that something changed about the cortex and it became possible to do these things for as I was that potential was already there but there wasn't the incentive to expand that capability and then use it wired to these social instincts and use it more I mean I would lean somewhat toward the latter I mean I think of mouse I has a lot of similarity

in terms of cortex as a human although there's that the Suzanna Herkila who's all were the um the the number of neurons skills better with weight with primate brains and it does with rodent brains right so yeah does that suggest that there actually was some improvement in the scalability of the cortex maybe maybe I'm not I'm not super deep on this there may there may have been yeah changes in architecture changes in the folding changes in neuron properties and stuff

that that somehow slightly tweak this but there's still a scaling that's right either way right so I was not saying there aren't something special about humans in the architecture of the learning

“subsystem at all but yeah I mean I think it's pretty widely thought that this is expanded but”

then the question is okay well how does that how does that fit in also with the steering subsystem changes and it the instincts that make use of this and route you to bootstrapped using this effectively but I mean just to say a few other things I mean so even the flybrain has some amount of for example and probably even even very far back um I mean I think you've read this this great book the brief history of intelligence right I think this is a really good book lots of AI researchers seeing

this is a really good book it seems like um yeah you have some amount of learning going back

all the way uh to anything that has a brain basically um you have something kind of like primitive

fish just like um and there's kind of these other branches birds maybe kind of reinvented something kind of cortex like but it doesn't have the six layers but they have something a little bit cortex like um so that that's some of those things um after reptiles and some sense birds and mammals both kind of made us up somewhat cortex like but differently organized thing but even a flybrain has like a sociative learning centers that um actually do things that maybe look a little

bit like this like thought assessor concept from from Beren's where there's like a specific dopamine signal to train specific subgroups of neurons in the fly mushroom body to associate different sensory information with am I going to get food now or going to get hurt now yeah brief tangent I remember reading in uh one blockpost that Beren millage wrote that the parts of the cortex which are associated with audio and vision have skill disproportionately between other primates and humans

whereas the parts associated say with odor have not and I remember him saying something like

“this is explained by that kind of data having worse scaling law properties but I think the”

and maybe he meant this but another interpretation of actually what's happening there is that these social reward functions that are built into the series of system needed to make use more of being able to see your elders and see what the visual cues are and what they're saying yeah in order to make a sense of these cues which guide learning you needed to activate these um yeah activate the vision and audio more than I mean there's all this stuff I feel like it's come up in in your

your shows before actually but like even like the design of the human eye where you have like the pupil and the light and everything like we're designed to be able to establish relationships based on joint eye contact and and maybe this came up in the sudden episode again remember but um yeah we we have to bootstrapped to the point where we can detect eye contact and where we can communicate

by language right and that's like what the the first couple years of life for right or trying to

do yeah okay I want to ask you about RL so currently the way these elements are trained you know they are if they solve the unit test or solve a math problem that whole trajectory every token in that trajectory is up weighted and what's going on with humans is there are there different types of model based versus model for either happening in different parts of the brain yeah I mean this this is another one of these things I mean again all my answers to these questions any specific

thing I say is all just kind of like directionally this is we can kind of explore around this I find this interesting maybe I feel like the literature points in these directions in some very broad way what I actually want to do is like go and map the entire mouse brain and like figure this out comprehensively and like make neuroscience the ground truth science so I don't know basically but

“but yeah I mean there so first of all I mean I think with Elia on the podcast I mean he was like”

it's weird that you don't use value functions right you use like the most dumbest form of RL basically of course there are these people are incredibly smart and they're optimizing for how

to do it on GPUs and it's really incredible whether achieving but like conceptually it's a really

dumb form of RL even compared to like what was being done in like 10 years ago right like even you know the Atari game playing stuff right was using like Q learning which is basically like it's a kind of temporal difference learning right and the temporal difference learning basically means you have some kind of a value function of like what action I choose now doesn't just tell me literally what happens immediately after this it tells me like what does the long run consequence of

that for my expected you know total reward or something like that and so you have value functions like the fact that we don't have like value functions at all is like in the LLM is like it's crazy

“I think I think because earlier said it I can say it you mean I know you know one one one”

one hundredth of what he does about AI but like it's kind of crazy that this is working but yeah I mean in terms of the brain well so I think there are some parts of the brain that are thought to do something that's very much like model free RL that's sort of parts of the basal ganglia sort of straight and basal ganglia they have like a certain finite like it is thought that they have a certain like finite relatively small action space and the types of actions

they could take first of all might be like tell the spinal cord or tell the brain stem and spinal

cord to do this motor action yes no or it might be more complicated cognitive type actions like tell the fellas to allow this part of the cortex to talk to this other part or release the memory this and the hippocampus and start a new one or something right but there's some finite set of actions that kind of come out of the basal ganglia and that it's just very simple RL so they're

algorithms um layer one thing on top of that is that some of the major work in neuroscience like

“Peter Diane's work and a bunch of work that is part of why I think DeepMind did the temporal”

difference learning stuff in the first place is they were very interested in neuroscience

and there's a lot of neuroscience evidence that the dopamine is giving this reward prediction error signal rather than just reward yes no you know how gazillion times up in the future it's a prediction error and that's consistent with like learning these value functions um so there's that and then there's maybe like higher order stuff so we have these cortex making this world model well one of the things the cortex world model can contain is a model of when you do and don't get

rewards right again it's predicting what the steering subsystem will do it could be predicting what the basal ganglia will do and so you have a model in your cortex that has more generalization and more concepts and all this stuff that says okay these types of plans these types of actions

will lead in these types of circumstances to reward so I have a model of my reward um some people

“also think that you can go the other way and so this is part of the inference picture this is”

idea of RL as inference you could say well conditional on my having a high reward sample of plan that I would have had to get there that's inference of the plan part from the reward part I'm clamping the reward as high and inferring the plan sampling from plans that could lead to that and so if you have this very general cortical thing it can just do if you have this like a general very general model based system in the model among other things includes plans and rewards

then you just get it for free basically so like in neural network of parlance there's a value head associated to the the omnidirectional inference that's happening yeah yeah there's or there's a value input yeah oh yeah and it it can predict one of one of the one of the almost sensory variables that can predict is is what rewards is going to get yeah but by this speaking of this thing about amortizing things um yeah obviously value is like amortized roll-out

looking upward yeah something like that yeah it's like a statistical average or prediction of it yeah right tangential thought uh you know the Joe Henrik and others have this idea that the way human societies have learned to do things is just like how do you figure out that you know

this kind of being which actually just almost always poisons you is edible if you do this

ten step incredibly complicated process and you want to wish if you fail at the being will be poisonous how do you figure out how to hunt this seal in this particular way with this particular weapon in this particular time of the year et cetera um there's no way but uh it's just like trying shit over generations and it's striking that this is actually very much like model for y'all happening at like a civilizational level um no not exactly the evolution is the simplest

algorithm in some sense right and if we believe that all this can come revolution like the outer loop can be like extremely not for sighted and yeah right um that that that's interesting just like uh hierarchies of evolution model for a culture evolution model for you so what does that tell you maybe that simple algorithms can just get you anything if you do it enough for right yeah yeah I don't know so but yeah so you you have like maybe this evolution model for you based on ganglia model for

a cortex model based culture uh model for you potentially um I mean there's like you pay attention to your elders or whatever or so there's maybe this like group selection or whatever of these things is like more model for you yeah but now I think culture well it stores some of the model yeah right so let's say you want to train an agent to help you with something like processing low in applications. Trading an agent to do this requires more than just

giving the model access to the right tools things like browsers and PDF readers and risk models. There's just a lot of tasks and knowledge that you can only get by actually working in an industry. For example certain low in applications will pass every single automated check despite being super risky every single individual part of the application might look safe but experience underwriters know to compare across documents to find subtle patterns that signal risk.

Liberal box has experts like this and whatever domain you're focused on and they will set up highly realistic training environments that include whatever subtle nuances and watchouts

“you need to look out for. Beyond just building the environment itself, Liberal box provides”

all this scaffolding you need to capture training data for your agent. They give you the tools to great agents performance and capture the video of each session and to reset the entire environment to a clean state between every episode. So whatever domain you're working in, Liberal box can help you train reliable, real-world agents. Learn more at labelbox.com/thorkash.

hardware in comparison to computers as exists now. So what I mean by this question is like if there's

“the algorithm with the algorithm just qualitatively for for much worse or much better if”

inscribed in the hardware of today and the reason to think it might like here's what I mean like

you know obviously the brain has had to make a bunch of trade officers are not relevant to competing hardware it has to be much more energetically efficient. Maybe as a result it has to run on slower speeds so that they're going to be smaller voltage gap. That's where the brain is runes at 200 hertz and has to run on 20 watts. On the other hand you know with like robotics we've clearly experienced that fingers are

way more than we can make motors so far as maybe there's something in the brain that is equivalent of like cognitive dexterity which is like maybe due to the fact that we can do unstructured sparsity we can co-locate the memory in the compute. Yes. Where does this all that are you like fuck we would be so much smarter if we didn't have to deal with these brains or you like oh I mean I think in the end we will get the best of both worlds right somehow right I think

I think an obvious downside of the brain is it cannot be copied you don't have you know external read-write access to every neuron and synapse whereas you do I can just edit something in the wait matrix right you know and Python or whatever you know and load that up and copy that in principle right so the fact that it can't be copied and kind of random access is like very annoying but otherwise maybe these are it like has a lot of advantages so or it

also tells you that you want to like somehow do the code design of the algorithm and the it maybe it even doesn't change it not much from all what we discussed but you want to somehow do this code design so yeah how do you do it with really slow low voltage switches that's going to be

“really important for the energy consumption the co-locating memory and compute so like I think”

that probably just like hardware companies will try to co-locate memory and compute they will try to use lower voltages allow some stochastic stuff there are some people that think that this like all this probabilistic stuff that we were talking about oh oh it's actually energy based models and so on is doing lot it is doing lots of sampling it's not just advertising everything that the neurons are also very natural for that because they're naturally stochastic and so you don't have to

do a random number generator and a bunch of Python code basically to generate a sample the neuron just generates samples and it can tune what the different probabilities are and so and like learn learn those tunings and so it could be that it's very code designed with like some kind of inference method or something yeah if you hilarious I mean the method you know all these people that folks make fun of on Twitter you know a young look cool a young look cool in the best jazos

whatever they're like no like we have maybe I don't know if that is actually one read of you know I haven't really worked on AI at all since LLMs you know took off so I'm just like out

of the loop but I'm surprised and I think it's amazing the scaling is working everything but

yeah I think the young the crew and Beth jazos are kind of onto something about the about the

“probabilistic models or at least possibly and in fact that's what you know all the neuroscientists”

and all the AI people thought like until 2021 right so there's a bunch of cellular stuff happening in the brain that is not just about neuron to neuron synaptic connections how much of that is functionally doing more work than the synapses themselves are doing versus it's just a bunch of collage that you have to do in order to make this anaptic thing work so the way you need to you know with a digital mind you can nudge the synapses right the parameter extremely easily but with

a cell to modulate a synapse according to the gradient signal it just takes all of this crazy machinery is it actually doing more than it takes extremely little to do so I don't know but I'm not a believer in the like radical like oh actually memory is not synapses mostly or like learning is mostly genetic changes or something like that I think it would just make a lot of sense I think

you put it really well for to be more like the second thing you said like let's say you want to

do weight normalization across all of the weights coming out of your neuron right or into your neuron well you probably have to go like somehow tell the nucleus about the cell and then how that kind of send everything back out to the synapses or something right and so there's gonna be a lot of cellular changes right or let's say that you know you just had a lot of plasticity and like your part of this memory and now that's got consolidated into the cortex or whatever and now we want to reuse you as

like a new one that can learn again it's gonna be a ton of cellular changes so there's gonna be tons of stuff happening in the cell but algorithmically it's not really adding something beyond these algorithms

and just find the weights and change them and it is a cell it just literally has to do all this

with molecular and machines itself without any central controller right it's kind of incredible

“there are some things that cells do I think that that seem like more convincing so”

in the cerebellum so one of these things the cerebellum has to do is like predict over time like predict what is the time delay you know let's let's say that you know I see a flash and then us you know some member milliseconds later I'm gonna get like a puff of air in my eye later something right the cerebellum can be very good at predicting what's the timing between the flash and the air puff so that now your eye will just like close automatically like the cerebellum

is like involved in that type of reflex like learned reflex and there are some cells in the cerebellum where it seems like the cell body is playing a role in storing that time constant changing that time constant of delay versus that all being somehow done with like I'm gonna make a longer ring of synapses to make that delay longer it's like no the cell body will just like store that time delay for you so there are some examples but I'm not a believer like out of the box in like

essentially this theory that like what's happening is changes and connections between neurons

“and that's like the main algorithmic thing is going on like I think that's a very good reason to”

still believe that it's that rather than some like crazy cellular stuff yeah going back to this whole perspective of like our our intelligence is not just this omnidirectional in friends thing that does a role model but really this system that teaches us what to pay attention to what are the important salient factors to learn from et cetera I want to see if there's some intuition you can drive from this but what different kinds of intelligence it might be like

so it seems like a GI or superhuman intelligence should still have this like ability to learn a world model that's quite general but then it might be incentivized to pay attention to different things that are relevant for what you know the modern post singularity environment how different to be expect different intelligence is to be basically yeah I mean I think one way of this question is like is it actually possible like make the paperclip maximizer whatever right if you make if you try to

make the paperclip maximizer does that end up like just not being smart or something like that because it was just the only reward function it had was like make paper clips if I channel Steve Burns Moore I mean I think he's very concerned that the sort of minimum viable things in the steering

“subsystem that you need to get something smart is way less than the minimum viable set of things”

you need for it to have human like social instincts and ethics and stuff like that so a lot of why you want to know about the steering subsystem is actually the specifics of how you do alignment essentially or what human behavior in social instincts is versus just what you need for capabilities we talked about it in a slightly different way because we were saying well in order for humans to like learn socially they need to make eye contact and learn from others but we

already know from hello lambs right but depending on your starting point you can learn language without that stuff right and so yeah and and so I think that it's probably is possible to make

like super powerful you know model based RL you know optimizing systems and stuff like that

that don't have most of what we have in the human brain reward functions and as a consequence might want to maximize paper clips and that's a concern yeah right but but you're pointing out that in order to make a competent paperclip maximizer yeah the kind of thing they can build the spaceships and learn the physics and whatever it needs to have some dries which elicit learning including say curiosity and exploration yeah curiosity and interest in others of so interest in

social interactions curiosity yeah but that that's that's pretty that's pretty minimal I think and it and that's true for humans right but it might be less true for like something that's already pre-trained as an LLM or something right and so so most of why we want to know the steering subsystem I think if I'm channeling Steve is alignment reasons yeah right how how confident are we that we even have the right algorithmic conceptual vocabulary to think about what the brain is doing

and what I mean by this is you know there was one big contribution to AI from neuroscience which was the site you have the neuron yeah we'll even fit you know 1950s just like the original contribution but then it seems like a lot of what we've learned afterwards about what the high-level algorithm the brain is implementing from the backdrop to if there's something in

that goes back up in the happening in the brain to always rewind doing something like CNN's to

TD learning and Bellman equations um actor critic whatever seems inspired by what is like we

we notice that's something in the brain also works that way yes so why not think there's more things

“like this where there may be yeah I think the reason that I'm not I think that we might be onto something”

is that like the AI's we're making based on these ideas or working surprising well if there's also a bunch of like just empirical stuff like convolutional neural nets and variants of convolutional neural nets I'm not for sure what the absolute latest latest but compared to other like models in computational neuroscience of like what the visual system is doing are just like more predictive right so you can just like score um even like pre-trained on like cat pictures and stuff CNN's

what is the representational similarity that they have on some arbitrary other image versus you know compared to the brain activations measured in different ways Jim de Carlos lab has the like brain score and like the AI models actually like there there there seems to be some relevance there in terms of like even like neuroscience is don't necessarily have something better than that so yes I mean that's just kind of recapitulating what you're saying

is that like the best computational neuroscience theories we have seem to have been like invented alright largely as a result of AI models and like fine things that work and so find back prop works and then say can we approximate back prop with cortical circuits or something and there's there's kind of been things like that no some people totally disagree with this right um so like Yuri Buzaki is a neuroscientist who has a book called the brain from inside out

we basically says like all our psychology concepts like AI concepts all the stuff is just like

made up stuff we actually have to do is like figure out what is the actual set of primitives that like the brain actually uses an our vocabulary is not going to be adequate to that we got to start with the brain and make new vocabulary rather than saying back prop and then try to apply that to the brain or something like that and you know he studies a lot of like oscillations and stuff in the brain as opposed to individual neurons and what they do and you know I don't know

“I think that there's a case to be made for that and from a kind of research program design perspective”

I think there's like one thing we should be trying to do is just like simulate a tiny where Maritime Ezebra fish like almost like as biophysical or like as as bottom up as possible like get connecto molecules activity and like just study it as a physical then chemical system and like look what it does but I don't know I mean just when I like I just feel like the AI is really good fodder for computational neuroscience like those might actually be pretty good models we should

look at that so I'm not a person who thinks that I think I I both think that there should be a part of the research portfolio that is like totally bottom up and not trying to apply our vocabulary that we learn from AI onto these systems and that there should be another big part of this that's kind of trying to reverse engineer at using that vocabulary or variance of that vocabulary and that we should just be pursuing both and and my guess is that the reverse engineering one is actually going to like

kind of workish or something like we do see things like TD learning which you know Sutton also invented right separately right that must be a crazy feeling to just like yeah that's great this is like equation I wrote down is like it seems like the dopamine is like doing some of that yeah so let me ask you about this you know you guys are finding different groups that are trying to yeah figure out what's up in the brain if we had a perfect

representation or you defined it of the brain why think of it actually let us figure out the answer of these questions we have neural networks which are way more interpretable not just because we understand what's in the weight matrices but because there are weight matrices there are these boxes with numbers in them right and even then we can tell very basic things we can kind of see circuits for yeah very basic pattern matching a falling one token with another right I I I feel like we

have we don't really have an explanation of why elements are intelligent just because there are

“yeah well I would there is somewhat I would somewhat dispute I think we have some architectural”

we have some description of what the LOM is like fundamentally doing and what that's doing is that I have an architecture and I have a learning rule and I have hyper parameters and I have initialization and I have training data but that those are things we learned from yeah because we built them not because we interpret them from seeing the way they built them which is the not the not the thing the connectome is like seeing the way what I think we should do is we should

describe the brain more in that language of things like architectures learning rules initializations rather than trying to find the golden gate bridge circuit and saying exactly how does this neuron actually you know that's going to be something incredibly complicated learned pattern yeah card recording and Tim Lilly crap have this paper from while ago maybe five years ago called what does it mean to understand a neural network or what what it mean to understand a neural

network and what they say is yeah basically that like you can imagine you train a neural network

to like compute the digits of pie or something well like some crazy you know it's like

thing you find predict stock price is a quite basically predict the really complex systems right computation you know computationally complete system like it predict I could train a neural network

to do cellular automata or whatever crazy thing and it's like we're never going to be able to

fully capture that with interpretability I think it's just going to just be doing really complicated computations internally but we can still say that the way it got that way is that it had an architecture and we gave it this training data and it had this loss function and so I wanted to describe the brain in the same way and I think that this framework that I've been kind of laying out is like we understand the cortex and how it embodies a learning algorithm

I don't need to understand how it computes golden game but if you can see all the neurons

“you have to connect them why does that teach you what the learning algorithm is?”

Well I guess there are a couple different views of it so it depends on this different parts of this portfolio so on the totally bottom up we have to simulate everything portfolio it kind of just doesn't you have to just like see what are the you have to make a simulation of the zebrafish right on or something and then you like see what are the like emergent dynamics in this and you come up with new names and new concepts and all that that's like that's like the most extreme

bottom-up neuroscience view but even there the connect home is like really important for doing that bottom biophysical or bottom-up simulation but on the other hand you can say well what if we can

actually apply some ideas from AI we basically need to figure out is it an energy-based model

or is it you know and they're more ties you know VIE type model you know is it doing back proper is it doing something else or the learning rules local level I mean if we have some repertoire of possible ideas about this can we just think of the connect home is a huge number of additional constraints that will help to refine to ultimately have a consistent picture of that.

“I think about this for the steering subsystems off to just very basic things about it how many”

different types of dopamine signal or of steering subsystem signal or thought assessor or so on how many different types of what broad categories are there like even there's a very basic information that there's more cell types in the hypothalamus and there are in the cortex like that's new information about how much structure is built there versus somewhere else yeah how many different dopamine neurons are there is the wiring between prefrontal and auditory the same as the wiring

between prefrontal and visual you know it's like the most basic things we don't know and the problem is learning even the most basic things by a series of bespoke experiments takes an incredibly long time or is just learning all that at once by getting a connect home is just like way more efficient.

What is the timeline on this because presumably the idea of this is to first

inform the development of AI you want to be able to figure out how we do the how we get AI to want to care about what other people think of his internal thought pattern but intrepresenters are making progress on this question just by inspecting normal neural networks there must be some future there make that you can do interpret on LLMs that exist yeah you can't do interpret on a hypothetical model-based reinforcement algorithm like the brain that we will eventually

converge to when we do a GI but you know what what timelines on AI do need for this research to

“be practical yeah relevant I think it's fair to say it's not super practical and relevant”

if you're in like a AI 2077 scenario you know and so like what science I'm doing now is not going to affect the science so like 10 years from now because what's going to affect the science of 10 years from now is the outcome of this like a 2027 scenario right it kind of doesn't matter that much probably if I have the connect home maybe it's slightly tweaks certain things but but I think there there's a lot of reasons to think maybe that we will get a lot out of this paradigm

but then the real thing the thing that is like the the like single event that is like transformative for the entire future or something type event is still like you know more than five years of air so it's right because like we haven't captured on my directional inference we haven't figured out the right ways to get a mind to pay attention to things in a way that I mean I would take the entirety of your like collective podcast with everyone as like showing like the distribution

of these things right I don't know right I mean what was carpet these timeline right you know what's dumbest is timeline right so these these didn't not everybody has a three-year timeline and so I think if there's different reasons and I'm curious what are mine I don't know I'm just watching your podcast I'm sorry I'm trying to understand the distribution I don't have a super strong claim that LLMs can't do it but it is a cross-legged efficiency

or is it the I think part of it is just it is weirdly different than all this brain stuff and so intuitively it's just weirdly different than all this brain stuff and I'm kind of waiting for like the thing that starts to look more like like I think they can give Alpha 0 and model based RL and all the these other things that were being worked on 10 years ago had been giving us the GPT-5 type capabilities then I would be like oh wow we're both in the right

right and now is like I don't know what exactly my data is looks pretty good but my prior is

“sort of weird so yeah so it's so I don't have a super strong opinion on it so I think there is a”

possibility that essentially all other scientific research that is being done is like not it is somehow updated but I don't put a huge amount of probability on that I think my timelines might be more in the like yeah a 10 year average range and if that's the case I mean I think there there yeah there is probably a difference of puno world where we have connectomes on hard drives and we have understanding of steering subsystem architecture we've

compared that that you know even the most basic properties of what are the reward functions cost function architecture et cetera of you know mouse versus a shrue versus a small primate et cetera it's a factor going 10 years I think it has to be a really big push like how much

funding how does it compare to where we are now it's like billion low billions dollars scale

funding we in a very concerted way I would say how much is this on it now well so so if I just talk about some of the specific things we have going so with connectomics so so E11 bio is kind of like the the our main thing on connectomics they are basically trying to make the technology of connectomic brain mapping several orders of magnitude cheaper so the welcome trust put out a report year or two ago that basically said to get one mouse brain the first mouse brain connect

home would be like several billion dollars you know billions of dollars project what E11 technology and sort of the suite of efforts in the field also are trying to get like a single mouse connectome down to like low tens of millions of dollars okay so that's a mammal brain right now a human brain is about a thousand times bigger so if a mouse brain you can get to 10

million or 20 million 30 million with technology you know if you just naively scale that okay

human brain is now still billions of dollars to just one do do one human brain can you go beyond that so can you get a human brain for like less than a billion but I'm not sure you need every

“neuron in a human brain I think we want to for example do an entire mouse brain and a human”

steering subsystem and the the entire brains of several different mammals with different social instincts and so I think that that with a bunch of technology push and a bunch of concerted effort can be done in the you know real significant progress if it's focused effort can be done in the kind of hundreds millions to low billions scale what is it definition of a connectome is it presumably it's not a bottom of biophysics model so is it just that if if it can

estimate the input output of a brain but like what is what is a level of structure so you can give different definitions and one of the things that's cool about so the kind of standard approach to connectomics use the electron microscope and very very thin slices of brain tissue and it's basically labeling the cell membranes are going to show up scatter electrons a lot and everything else is going to scatter electrons less but you don't see a lot of details of the molecules which types

of synapses different synapses of different molecular combinations and properties e11 and some other research in the field has switched to an optical microscope paradigm with optical the photons don't damage the tissues so you can kind of wash it and look at fragile gentle molecules so so with the e11 approach you can get a quote unquote molecularly annotated connectome so that's not just who is connected to who by some kind of synapses but what are

the molecules that are present at the synapses what type of cell is that so molecularly annotated connectome that's not exactly the same as having the synaptic weights that's not exactly the same as being able to simulate the neurons and say was the functional out functional consequence of having these molecules and connections but you can also do some amount of activity mapping and try to correlate structure to function yeah so interesting training ML model to basically

predict the activity from the connectome what are the lessons to be taken away from the human genome project because when we could look at it is that it was actually a mistake and you shouldn't

“spend whatever billions of dollars getting one yeah genome map rather you should have just”

invested in technologies which have now now lost a map genome is for hundreds of dollars yeah well yeah so George Church was my it was my PhD advisor and and basically yeah I mean what he's pointed

out is that yeah it's 3 billion or something roughly one dollar per base pair for the first genome

and then the National Human Genome Research Institute basically structured the funding process rights and they got a bunch of companies competing to lower the cost and then they cost dropped like a million fold in ten years and because they changed the paradigm from kind of macroscopic kind of chemical techniques to these individual DNA molecules make a little cluster of DNA molecules and microscope and you would see just a few DNA molecules at a time

on each pixel of the camera would basically give you a different in parallel looking at different fragments of DNA so you paralyzed the thing by like millions fold and that's what reduced the

to optical connectomics potentially even future types of connectomics technology we think

“there should be similar pattern as that's why 11 with the focus research organization”

started with technology development rather than starting with saying we're going to do a human brain or something let's just brute force it we said let's get the cost down with new technology but then you still it's still big thing even with new next generation technology you still need to spend hundreds of millions on data collection yeah is this going to be funded with philanthropy by governments by investors this is very TBD and very much evolving in some sense as we speak

I'm hearing some rumors going around of connectomics related companies potentially forming but so so so far 11 has been philanthropy the National Science Foundation just put out this

call for it for tech labs which is basically somewhat of it is kind of fro inspired or related

I think you could have a tech lab for actually going and mapping the mouse brain with us and that would be sort of philanthropy plus government still in a non-profit kind of open source framework but can can companies accelerate that can you read credibly link connectomics to AI in the context of a company and get investment for that it's like possible I mean the cost of training these ads is increasing so much if you get like tell some story yeah like

not only are going to figure out some safety thing right but in fact we will once we do that we'll also be able to tell you how we are I mean all these issues like go to these AI labs and just be like give me one 100th of your projected budget 20 years I sort of tried a little bit like like seven or eight years ago and there was not a lot of interest and maybe

“now there would be but yeah I mean I think all the things that we've been talking about like”

I think it's really fun to talk about but it's ultimately speculation what is the actual

reason for the energy efficiency of the of the brain for example right is it doing real inference or immortalize inference or something else like this is all going to be all this all answerable by neuroscience it's going to be hard but it's actually answerable and so if you can only do that for low billions of dollars or something to really comprehensively solve that it seems to me in the grand scheme of trillions of dollars of GPUs and stuff it actually

makes sense to do that investment but and in any of us there's also just there's been many lots that have been launched in the last year where they're raising on the valuation billions yes where things which are quite credible but are not like our ARR next quarter is going to be whatever it's like we're going to discover materials and dot dot dot dot right yes moonshot startups or billion dollar I billion ARR back startups moonshot startups I see isn't kind of on a

continuum with froze yeah froze our way of channeling philanthropic support ensuring that it's open source public benefit very other things that that may maybe properties of a given fro but yes billion ARR back startups if they can target the right science the exact right science I think there's a lot of ways to do

moonshot neuroscience companies that would never get you the connect home you're like oh we're

going to upload the brain or something but never actually get the the mouse connect home or something

“these fundamental things that you need to get to to ground truth the science there are lots of ways”

to have a moonshot company kind of go wrong and not do the actual science but they're also maybe ways to have companies or or big corporate labs get involved and actually do it correctly yeah this uh this brings to mind an idea that you had in a leisure game five years ago about yeah doing explain behavior cloning on right yeah I mean actually this is funny because I think that the first time I saw this idea it was I think it actually might have been in a blog post by Gordon

there's always there's always a Gordon blog post and there are now academic research efforts and some amount of emerging company type efforts to try to do this so um yeah so normally let's let's say I'm training an image classifier something like that I show it pictures of cats and dogs or whatever and they have laid the label cat or dog and I have a neural network supposed to predict the label cat or dog or something like that um that is a limited amount of information

per label that you're put again it's just cat or dog what if I also had predicts what is my neural activity pattern when I see a cat or when I see a dog and all the other things um if you add that as like auxiliary loss function or an auxiliary prediction task does that sculpt the network to know the information that humans know about cats and dogs and to represent it in a way that's consistent with how the brain represents it and the kind of representation kind of dimensions

or geometry of of how the brain represents things as opposed to just having these labels does that let it generalize better does that let it have just richer labeling and of course that's like

with you know scale AI or whatever can do this it is harder to generate lots and lots of brain activity patterns that correspond to things that you want to train the AI to do but again this is just a technological limitation of neuroscience if every iPhone was also a brain scanner you know you would you would not have this problem and be would be training AI with the brain signals then it's just the order in which technology is developed is that we got GPUs before

we got portable brain scanners or whatever right and that kind of thing what is the ML and the like what you'd be doing here is when you just still models you're still looking at the the final layer like the the log props across across across if you do distillation of one model into another

“that is a certain thing you're just trying to copy one model into another yeah I think that we don't”

really have a perfect proposal to like distill the brain I think to distill the brain you need like a much more complex brain interface like maybe you could also do that you could make surrogate models Andreas told you some people like that are doing some amount of neural networks surrogate models of brain activity data instead of having your visual cortex

do the computation just have the surrogate models you're basically distilling your visual cortex into a neural network to some degree that's a kind of distillation this is doing something a little different this is basically just saying I'm adding an auxiliary I think of his regularization or I think of it as adding an auxiliary loss function that's sort of smoothing out the prediction task

to also always be consistent with how the brain represents it like what exactly it might

help predict like adversarial examples for example right or you're predicting the internal state of the brain yes so in addition to predicting the label the vector of labels like yes cat not dog yes you know not boat you know one shot vector or whatever of one hot vector of yes it's cat instead of these gazillion other categories let's say in this simple example you're also predicting a vector which is like all these brain signal measurements right yeah interesting and so

weren't anyway had this long ago blog post of like oh this is like an intermediate thing this like we talk about whole brain emulation we talk about agi we talk about brain computer interface we should also be talking about this like brain augmented brain data augmented thing this trained on all your behavior but is also trained on like predicting some of your neural patterns right you're saying the learning system is already doing this really steering system yeah and our learning

system also has predict the steering subsystem as a linksillary task yeah yeah and that helps the steering subsystem now the steering subsystem can access that predictor and build a cool reward function using it yes okay separately you're on the board for of lean which is this formal formal math language that the mathematicians used to prove theorems and so forth and obviously there's a bunch of conversations right now but math yeah automating math what's

“your take yeah well I think that there are parts of math that it seems like it's pretty well”

on track yeah to to automate and that has to do with like so so first of all so so lean so

lean had had been developed for a number of years at Microsoft and other places has become one of the convergent focus research organizations kind of drive more engineering and focus on to it so lean is like this language programming language where if you instead of expressing your math proof on pen and paper you express it in this programming language lean and then at the end if you do that that way it is a verifiable language so that you can basically click verify and lean will tell you

whether the conclusions of your proof actually follow perfectly from your assumptions of your proof so it checks whether the proof is correct automatically um like by itself this is useful for mathematicians collaborating and stuff like that like if I'm some amateur mathematician I want to add to a proof you know Terry Tao is not going to like believe my result but if lean says it's correct it's just correct so he makes it easy for like collaboration to happen but it also makes it easy for

correctness of proofs to be an RL signal in very much the RL VR you know it's like a perfect math proofing is now formalized math proofing is a formal means it's like expressed in something like lean and verifiable mechanically verifiable um that becomes a perfect RL VR you know task

“yeah and I think that that is going to just just keep working it seems like is the couple billion”

dollars you know these one like billion dollar valuation company harmonic based on this

alpha proof is based on this um a couple of their emerging really interesting companies I think that this problem of like RL VRing the crap out of math proving is basically going to work

that we have alpha go or what have you that can search for you know ways of playing the game of go and with that verifiable signal works so does this like solve math there is still the part that has to do with conjecturing new interesting ideas there's still the kind of conceptual organization of math of what is interesting how do you come up with new theorem statements

in the first place or even like the very high level breakdown of what strategies you use to

“do proofs um I think this will shift the burden of that so that humans don't have to do a lot”

of the mechanical parts of math validating lemmas and proofs and checking if the statement of this in this paper is exactly the same as that paper and stuff like that it would just that will just work you know if you really think you're we're going to get all these things we've been talking about really agi it would also be able to make conjectures and you know Benjio has like a paper is more like theoretical paper they're probably a bunch of other papers emerging

about this like is there like a loss function for like good explanations or good conjectures that's like a pretty profound question right a math a really interesting math proof or statement might be one that can compresses lots of information about other you know has lots of implications for lots of other theorems otherwise you would have to prove those theorems using long complex passive inference here if you have this theorem this theorem is correct you have short passive

inference to all the other ones and it's a short compact statement so it's like a powerful

explanation that explains all the rest of math and like part of what math is doing is like making these compact things that explain the other things so the kogo more complexity of this statement or something yeah generating all the other statements given that you know this one or stuff like that or if you add this how does it affect the complexity of the rest of the kind of network of proofs so can you like make a loss function that adds oh I want this proof to be a really highly powerful

“proof I think some people are trying to work on that so so maybe you can automate the creativity part”

if you had true age I it would do everything human can do so it would also do the things that them create of mathematicians do but um but we're barring that I think just our LVRing the crap out of proofs

well I think that's going to be just a really useful tool for mathematicians so they're going to

accelerate math a lot and change it a lot but not necessarily immediately change everything about it well we get you know mechanical proof of the reman hypothesis or something like that or things like that maybe I don't know I don't know enough details of how hard these things are to search for I'm not sure anyone can fully predict that just as we couldn't exactly predict when go would be solved or something like that um and I think it's going to have lots of really cool applied applications

so um one of the things you want to do is you want to have probably stable secure, unhackable etc software so you can write math proofs about software and say this code not only does it pass these unit tests but I can mathematically prove that there's no way to hack it in these ways or no way to mess with the memory or this type of things that hackers use or it has these properties it can use the same lean and same proof

“to do formally verified software I think that's going to be a really powerful piece of cybersecurity”

that's relevant for all sorts of other AI hacking the world stuff and that yeah if you can prove reman hypothesis you're also going to be able to to prove insanely complex things about very complex software and then you'll be able to at the LLM synthesize me a software that is uh I can prove is correct right what why hasn't approval um programming language taken off as a result of all the ones you would think that this is starting to yeah I think

it's starting to I think that one one challenge and we are actually incubating a potential focus research organization on this is the specification problem so mathematicians are kind of know what interesting theorems they want to formalize um if I have like some code let's say I have some code that like is you involved in running the power grid or something it has some security properties well what is the formal spec of those properties the the power grid engineers just made this thing

but they don't necessarily know how to lift the formal spec from that and it's not necessarily easy to come up with the spec that is the spec that you want for your code people aren't used to coming up with formal specs and they're not a lot of tools for it so you also you have like this kind of user interface plus AI problem of like what security spec should I be specifying is this the spec that I wanted so there's a spec problem um and it's just been really complex and hard

but but it's only just in the last very short time that that the LLM's are able to generate you know verifiable proofs of you know things that are useful to mathematicians starting to be able to do some amount of that for for software verification hardware verification

it's possible that it just flips the tie that formal methods based this whole field of formal

“methods are formal verification provable software which is kind of this weird almost like back”

water of more like theoretical part of programming languages and stuff very academically flavored often although there was like this DARPA program that made like a provably secure like quadcopter helicopter and stuff like that so it's a cure against like what is the property that is exactly proved um and I'm not for that particular project but it doesn't general like us so what what because obviously things malfunction for all kinds of reasons you could say that what's going on in this part of the

memory over here which is supposed to be the part the user can access can't in any way affect what's going on in the memory over here or something like that or yeah things like that yeah got it yeah

so there's there's two questions one is how useful is this and two is like how satisfying

as a as a mathematician would it be um and the fact that there's this application towards proving that software has certain properties or hardware and properties obviously like if that were that would obviously be very useful but from a pure like are we going to figure out mathematics right um yeah is there is your sense that there's something about finding that one construction cross maps to another construction in a different domain or finding that oh this like

lemma is if you reconfigure it like if you redefine this term it's still like kind of satisfies what I meant by this term but it no long the a kind of example that previously knocked it down no longer applies like that kind of dialectical thing that happens in mathematics well it's software like replace that yeah I know like how much of the value of this sort of pure mathematics just comes from actually just coming up with entirely new ways of thinking about a problem yeah like mapping

“into a totally different representation and do you have examples of I don't know I think if it is”

I think it maybe a little bit like the uh it when every everybody had to write assembly code or something like that just like the amount of fun like cool start-up should have got created it was like a lot less or something like and so it was just like less people could do it progress was more grinding and slow and lonely and so on um you had more false failures because you didn't get something

about the assembly code right rather than the essential thing of like was your concept writes um harder

to collaborate and stuff like that and so I think it will like be really good um there is a some worry that by not learning to do the mechanical parts of the proof that you fail to generate the intuitions that inform the more conceptual parts create a part right yeah it's the same with assembly and right and and so so what point is that applying is vibe coding or people not learning computer science right or actually are they like vibe coding and they're also simultaneously

looking at at the LOM with like explaining them these abstract computer science concepts and it's all just like all happening faster their feedback loop is faster in their learning way more abstract computer science and algorithm stuff because they're vibe coding you know I don't know um that it's not obvious that might be something the user interface and the human infrastructure around it but I guess there's some worry that people don't learn the mechanics and therefore don't build

like the grounded intuitions or something but my hunch is it's like super positive exactly on net how useful that will be or how much overall math like breakthroughs or like math breakthroughs even that we care about what happened I don't know I mean one other thing

“that I think is cool is actually the accessibility question it's like okay that sounds a little bit”

corny okay out more people can do math but but who cares but I think there's actually lots of people that like could have interesting ideas like maybe the quantum theory of gravity or something like yeah one of us will come up with the quantum theory of gravity instead of like a car carrying physicist in the same way that Steve Burns is like reading the neuroscience literature and he's like hasn't been in a neuroscience lab that much but he's like able to synthesize

across the neuroscience literature be the learning subsystems during subsystem does this all make sense he's you know it's kind of like he's an outsider at neuroscientists in some ways can you have outsider you know string theorists or something because the math is just done for them by the computer and does that lead to more innovation in the string theory right maybe yes interesting so okay so if this approach works and you're right that elephants are not the final paradigm

and suppose it takes a at least 10 years to the final paradigm yeah in that world there's this fun sci-fi premise where you have the turns towel to the head of tweet or he's like these these models were like automated cleverness but not automated intelligence and you can quibble with the definitions there but yeah if you have automated cleverness and you have some way of filtering which if you can formalize and prove things that the elements are saying

yes and so how what are the domains of the world which could be put in this

provable symbolic representation yeah furthermore okay so in the world where I just ages super far away maybe makes sense to like literally turn everything the elements ever do or almost every time they do into like super provable statements and so elephants can actually build on top of each other because everything do is like super provable yeah maybe this is like just necessary because we have billions of intelligence is running around

“even if they are super intelligent in the only way the future ages of authorization to collaborate”

with each other is if they can prove each step yeah and they're just like brute force turning out that this is what the Jupiter brains are doing it's a universal language it's provable and it's also provable from like are you trying to exploit me or are you sending me some yeah some message that's actually trying to like sort of hack into my my brain effectively are you trying to socially influence me are you actually just like setting me just the

information that I need and no more right for this and yes with Davi Dodd it was like this program director at Arya now in the UK I mean he has this whole design of a of a kind of ARPA style program of sort of safeguarded AI that very heavily leverages like provable safety properties and can you apply proofs to like can you have a world model but that world model is actually not specified just in neuron activations but it's specified in you know equations

those might be very complex equations but if you can just get insanely good at just auto proving these things with cleverness auto cleverness can you have you know explicitly interpretable

world models you know um as opposed to neural net world models and like move back basically the

symbolic method just because you can you can just have insane amount of ability to prove things yeah I mean that's an interesting vision I don't know how you know in the next 10 years like

“whether that will be the vision that plays out but I think it's really interesting to think about”

yeah and even for math I mean I think territory towers like doing some amount of stuff where it's like it's not about whether you can prove the individual theorems it's like let's prove all the theorems on mass and then it's like study the properties of like the aggregate set of proved theorems right which other ones they got proved in which other ones they didn't okay well that's like the landscape of all the theorems instead of one theorem at a time right I see

speaking of symbolic representations one of the equation I was meaning to ask you is how does the brain represent the world model like obviously nets out of neurons but at I don't mean sort of extremely functionally I mean sort of conceptually is it in something that's analogous to the hidden state of a neural network where is it something that's closer to a symbolic language we don't know I mean I think there's there's some amount of study of this I mean

there's there's these things like you know face patch neurons that represent certain parts of the face that geometrically combined in interesting ways that's sort of with geometry and vision is that true for like other more abstract things there's like this idea of cognitive maps like a lot of stuff that a rodent hippocampus has to learn is like place cells and like where is the rodent going to go next and is it going to get a reward there is like very geometric and

like do we organize concepts with like a abstract version of a spatial map there's some questions of can we do like true symbolic operations that can I have like a register and my brain that copies a variable to the another register regardless of what the content of that that

variable is that's like this variable binding problem and basically I just don't I don't know if

we have that like machinery or if it's like more like cost functions and architectures that like make some of that approximately emerge but maybe it would also emerge in a neural net there's a bunch of interesting neuroscience research trying to study this what what the representations look like what was your hunt yeah my huntress is going to be a huge mess and we should look at the architectures the lost functions in the learning rules and we shouldn't really

I don't expect it to be pretty in there yeah which is it is not a symbolic language not yeah probably it probably it's not that symbolic yeah but but but other people think very differently you know yeah other random questions it's meeting and binding yep but what is up with feeling like there's an experience that it's like both all the part of your brain which are modeling very different things have different drives feel like at least presumably feeling there's an

experience happening right now and also that across time you feel like what is I I'm pretty much out of loss on this one I don't know I mean Max Hodak has been really giving talks about this recently he's another really hardcore neuroscience person neurotechnology person and the thing I mentioned with Doris so it maybe also sounds like it might have some touching on this question but

“yeah I think this I have I don't think anyways any idea it might even involve new physics”

like it yeah another question which might not have an answer yet what so continual learning is that the product of some thing extremely fundamental the level of even the learning algorithm

way there's a training period and you freeze the way it's um and so you just need this

active inference or some other learning rule in order to do learning or do you think it's more a matter of architecture and how is memory exactly stored and is it like what kind of

“associated memory you have basically yeah so continual learning um I don't know I think that there's”

probably things that there's probably some at the architecture level there's probably something interesting stuff for the hippocampus is doing um and people have long thought this um what kinds of sequences of storing how is it organizing representing that how is it replaying it back what is it replaying back um how is it exactly how that memory consolidation works in I was sort of training the cortex using replays or or or it's memories from the hippocampus or

something like that um there's probably some of that stuff there might be multiple timescales of

plasticity or sort of clever learning rules um that can kind of I don't know can sort of simultaneously kind of be storing sort of short-term information and also doing back prop with it and there are maybe doing a couple of things you know some fast weight plasticity and some slower plasticity at the same time or synapses that have many states I mean I don't know I mean I think

“that from a neuroscience perspective I'm not sure that I've seen something that's super clear”

on what continual learning what causes it except maybe to say that this system's consolidation idea of sort of hippocampus consolidating cortex like some people think is a big piece of this and we don't still fully understand the details yeah it's speaking of fast weights is there something in the brain which is the equivalent of this distinction between parameters and activations that we see in neural networks and specifically like in transformers we have this

ideas like some of the activations are the key and value um vectors of previous tokens that you you build up over time and there's like the so-called the fast weights that you whenever you have a new token you query them against these um you query it as these activations but you also obviously care query them against all the other parameters in the network which are part of the actual built-in weights is there's some such distinction that's analogous

I don't know I mean we definitely have weights and activations whether you can use the activations in these clever ways um different forms of like actual attention like attention in the brain

“is that based on I'm trying to pay attention I think there's several probably several”

different kinds of like actual attention in the brain I want to pay attention to this area of visual cortex I want to pay attention to this the content in other areas that is triggered by the content in this area right attention this just based on kind of reflexes and stuff like that so I don't know I mean I think that there's not just the cortex there's also the thalamus the thalamus is also involved in kind of somehow relaying or gating informations there's

cortical cortical connections there's also some amount of connection between cortical areas that goes through the thalamus is it possible that this is doing some sort of matching or kind of constraints as faction or matching across you know keys and about you know keys over here and values over there is it possible that it can do stuff like that maybe I don't know this is all part of what's the architecture of this cortical thalamus yeah system I don't know I don't know

how transformer like it is or if there's anything analogous to like that attention be interesting to find out we got to give you a billion dollars so we can now you can want to broadcast again and we'll be great to tell you how exactly the manner mostly I just do data collection it's like really really really unbiased data collection so all the other people can figure out these questions yeah maybe the final question to go off on is what was the

most interesting thing you learned from the gap map and you want to explain with the gap map is so the gap map so in the process of incubating and coming up with these focus research organizations these sort of nonprofits start up like moon shots that we've been getting for anthropists and now government agencies to fund we talked to a lot of scientists and some of the

scientists were just like here's the next thing my graduate student will do here's what I find

interesting exploring these really interesting hypotheses spaces like all the types of things we've been talking about and some of them are like here's this gap um I need this piece of infrastructure which like there's no combination of the grad students in my lab or me loosely collaborating with other labs with traditional grants that could ever give me that I need to have like an organized engineering team that like builds you know the the mini miniature equivalent of the Hubble Space Telescope

and if I can build a Hubble Space Telescope then like I will unblock all the other researchers in my field or some like path of technological progress in the way that the Hubble Space Telescope made lifted the boats improve the life of every astronomer but wasn't really an astronomy discovery in itself it was just like you had to put this giant mirror in space with the CCD camera and like organize all the people and engineering and stuff to do that um so some of the things we talked

those things and it's like we taught a gap map um it's I think it's actually more like a fundamental

“capabilities map like what are all these things like mini Hubble Space Telescopes um and then we kind”

of organize that into gaps for like helping people understand that or like search that and what was the most surprising thing you found so I mean I think I've talked about this before but I think it one thing is just like kind of like the overall size or shape of it or something like that is like it's like a few hundred fundamental capabilities so each of these was like a deep tech startup

size project that's like only a few billion dollars or something like you know that was

each one of those was a series a that's only like not you know it's not like a trillion dollars to solve these gaps it's like lower than that so that's like one maybe we assumed that and we also

“came to you know that's what we got it's not really comprehensive it's really just a way of summarizing”

a lot of conversations we've had with scientists um I do think that in the aggregate process like things like lean are actually like surprising because I did start from sort of neuroscience and biology you know like very obvious that there's sort of like these omics we need genomics but you also need connect

omics and you know we can engineer equal i but we also need to engineer the other cells and like

there's like somewhat obvious parts of biological infrastructure I did not realize that like math proving infrastructure like was a thing and so um and that was kind of like immersion from trying to do this so I'm looking forward to seeing other other things where it's like not actually this like hard intellectual problem to solve it it's maybe the kind of the slightly equivalent of AI researchers just needed GPUs or something like that and focus and and and really good pie torch code to like

start doing this like what is the full diversity of fields in which that exists um we've even now found

and which are the fields that do or don't need that so fields that have had

gazillions of dollars of investment do they still need some of those do they still have some of those gaps or is it only more like neglected fields um we're even finding some interesting ones in actual astronomy actual telescopes that have not been explored because maybe because of that kind of

“if you're getting above a critical mass size project then you have to have like a really big”

project and that's a more bureaucratic process with the federal agencies yeah I guess you just kind of need scale in every single domain of science these days yeah I think you need scale in many of the domains of science and that does not mean that the low scale work is not important does not mean the kind of creativity, the serendipity etc each student pursuing a totally different direction or thesis that you see in universities is not like also really key but yeah I

think we need some amount of scalable infrastructure is missing in essentially every area of science even math which is crazy because math mathematicians I thought just needed whiteboards right but they actually need lean they actually need verifiable programming languages and stuff like like I didn't know that all right cool I don't know this is super fun there's going on thank you so much for people find your stuff pleasure the easiest way now my my out-of-marble sound that org website is currently

down I guess but you can find conversion research.org can can make too a lot of the stuff we've been doing yeah and then you have a great blog, a long-term role sign yes one just you're not science yes on WordPress yeah cool thank you so much pleasure yeah hey everybody I hope you enjoyed that episode if you did the most helpful thing you can do is just share it with other people who think might enjoy it's also helpful if you leave a rating or comment on whatever platform you're listening

on if you're interested in sponsoring the podcast you can reach out at barcash dot com slash advertise otherwise I'll see you at the next one