Over lunch

Over lunch, my sister called me to share her fears about the world and the direction of AI development. As a software engineer in Stockholm, she is seeing first-hand how the industry is changing – how AI-generated, customised code is rapidly replacing open source and human intent.

Always the optimist, I try to stay sane by taking Caia to Tylösand. Yet, with companies like Anthropic having made AGI their explicit near-term target, I cannot deny that I am scared too. Even if we somehow avoid turning all of the solar system into “computronium”, the consequences of an intelligence explosion are, to a large extent, beyond human imagination.

For now, our best hope seems to be that human-level intelligence turns out to be a much harder problem than the past five years of AI development have given us reason to believe. Either that – or that something is in fact “wrong” with reality at a much more fundamental level (think Fermi’s paradox).