The "Analysis and criticism" section of the Wikipedia page for ECREE has a useful summary of why the ridiculously subjective "extraordinary" is often unhelpful:
In the case of Atlas, I'm on team Fermi - meaning the claim that a 13 billion year-old galaxy is *not* awash in artifacts from prior civilizations strikes me as extraordinary. That doesn't mean I should require extraordinary evidence for the claim that Atlas is just a comet. Claims require evidence. That's it. And we should take a Bayesian approach to seeing which explanations look like the best fit for the available evidence so we can design more informative experiments. "Just a comet" has turned out to be an extraordinarily crappy fit for the available evidence - so we should look at X-rays, far-ultraviolet, radio, potentially data-bearing modulation of polarized light, and the orbit of Mars.
A lot of my time in the lab is spent in the loop where I do an experiment, the results turn out to be something none of my starting hypotheses anticipated, and then it's back to the drawing board to develop better experiments. Sometimes the claim ends up being something I totally failed to imagine at the beginning - meaning there was initially no way of anticipating impact and likelihood when I first committed to the testing. It's why you and I disagree about whether X-prize-like mechanisms are a sensible way to fund science. If you don't throw a lot of darts at stuff that just looks purely blue sky at the beginning then science will gradually grind to a crawl. Over to Henry George:
"The fact is that the work which improves the condition of mankind, the work which extends knowledge and increases power and enriches literature, and elevates thought, is not done to secure a living. It is not the work of slaves, driven to their task either by the lash of a master or by animal necessities. It is the work of men who perform it for their own sake, and not that they may get more to eat or drink, or wear, or display. In a state of society where want is abolished, work of this sort could be enormously increased."
Although I'm not entirely onboard with your formula of impact x probability = value as a strategy for prioritizing science funding, it does seem like a sensible approach to engineering and commerce. Or other practical things, such as planetary defense. The hypothesis that Atlas is a neutron star is a classic black swan - improbable, but with profound impact if true.
And by "impact" I mean an actual shitstorm of stuff falling from the sky in coming years, due to disruption of the orbits of outer solar system objects. Kind of like the old Nemesis hypothesis:
I guess passage of a high-mass object could hypothetically explain the unusually intense solar storms we've been experiencing in the past week?
A high-mass object could also cause earthquakes and altered ocean tides. Unfortunately, the InSight lander - which was designed to measure marsquakes - lost power due to dust on its solar panels. But I guess the observatories orbiting Mars could look for evidence of recent quakes, in the form of new fissures or shifting rocks. Viva cheap experiments!
We just need to get rid of the word "extraordinary" when describing hypotheses. It's not helping anything.
(impact if true) * (probability of it being true) = value of testing
Potential high value outcomes are worth high costs of testing, but if you can find a cheap way to test it, even better.
The "Analysis and criticism" section of the Wikipedia page for ECREE has a useful summary of why the ridiculously subjective "extraordinary" is often unhelpful:
Extraordinary_claims_require_extraordinary_evidence#Analysis_and_criticism
In the case of Atlas, I'm on team Fermi - meaning the claim that a 13 billion year-old galaxy is *not* awash in artifacts from prior civilizations strikes me as extraordinary. That doesn't mean I should require extraordinary evidence for the claim that Atlas is just a comet. Claims require evidence. That's it. And we should take a Bayesian approach to seeing which explanations look like the best fit for the available evidence so we can design more informative experiments. "Just a comet" has turned out to be an extraordinarily crappy fit for the available evidence - so we should look at X-rays, far-ultraviolet, radio, potentially data-bearing modulation of polarized light, and the orbit of Mars.
A lot of my time in the lab is spent in the loop where I do an experiment, the results turn out to be something none of my starting hypotheses anticipated, and then it's back to the drawing board to develop better experiments. Sometimes the claim ends up being something I totally failed to imagine at the beginning - meaning there was initially no way of anticipating impact and likelihood when I first committed to the testing. It's why you and I disagree about whether X-prize-like mechanisms are a sensible way to fund science. If you don't throw a lot of darts at stuff that just looks purely blue sky at the beginning then science will gradually grind to a crawl. Over to Henry George:
"The fact is that the work which improves the condition of mankind, the work which extends knowledge and increases power and enriches literature, and elevates thought, is not done to secure a living. It is not the work of slaves, driven to their task either by the lash of a master or by animal necessities. It is the work of men who perform it for their own sake, and not that they may get more to eat or drink, or wear, or display. In a state of society where want is abolished, work of this sort could be enormously increased."
Although I'm not entirely onboard with your formula of impact x probability = value as a strategy for prioritizing science funding, it does seem like a sensible approach to engineering and commerce. Or other practical things, such as planetary defense. The hypothesis that Atlas is a neutron star is a classic black swan - improbable, but with profound impact if true.
And by "impact" I mean an actual shitstorm of stuff falling from the sky in coming years, due to disruption of the orbits of outer solar system objects. Kind of like the old Nemesis hypothesis:
https://en.wikipedia.org/wiki/Nemesis_(hypothetical_star)
I guess passage of a high-mass object could hypothetically explain the unusually intense solar storms we've been experiencing in the past week?
A high-mass object could also cause earthquakes and altered ocean tides. Unfortunately, the InSight lander - which was designed to measure marsquakes - lost power due to dust on its solar panels. But I guess the observatories orbiting Mars could look for evidence of recent quakes, in the form of new fissures or shifting rocks. Viva cheap experiments!