I would never trust an AI to make a scientific conclusion, but they're very good at some things:
- Plotting with natural language: "Create a scatter plot of column A versus column B, log scale Y and use stars sized by column C." Every plotting library (except maybe ggplot2 / plotnind) sucks, and the boilerplate never gets less tedius.
- Literature search, e.g. "Have there been any peer reviewed papers where they applied X method to Y domain"
- Complex visualization, e.g. "Make a self contained HTML file with a 3D interactive visualization of this graph. Add ball and spring position optimization and let me move nodes"
- Trying random stuff that would normally be too time consuming to be worth it, eg "Here is a dataset. Try regression methods A, B, and C and report a table." In this case, usually it will screw up and overfit, but it's a good smoke test.
Note how in all of these, it's being used for tedious things that provide some value, but ultimately they are only providing information to make a decision on.
Granted, I would like to have a research assistant I could train, but if your company will not create a position then there is not much you can do.
Yes to all but I have trouble using them effectively for my research side projects or Kaggle competitions. Claude Code/Codex designs and runs experiments quickly, invalidates my initial idea using some obscure criteria than changes direction, and after long hours and days of experiments we end up with poor results. There must be a better agentic workflow for research.
> invalidates my initial idea using some obscure criteria than changes direction
You might be giving it too much leeway.
Opus and Fable are more trustworthy, but even today I find they very quickly go off the rails for research tasks if I don't give them short chunks of work.
They're not ready for prime time automation of research (and before someone mentions the math results, that required absurd amounts of computation time and has immediate feedback in the form of Lean)
AI in science is a lot different than AI in progamming in some ways. It seems pretty harmless in programming a lot of the times because it's used to speed up rote tasks like writing boilerplate. But in science, if it does give scientists a boost in discovery, I think it's likely to lead to an even more cavalier attitude about the power that such discoveries can have, and thus lead to much more abuse of power by the wealthy and malicious actors.
> Using privacy-preserving methods, we study 207,000 Claude and Claude Code research sessions to understand how Claude is used across scientific fields and research tasks.
Did they get permission to use researchers' data or did they monitor and extract data without consent? Considering the kerfuffle OpenAI generated with their math proofs, you'd think Anthropic would take a hint and be clear that they did the research with consent, though that assumes they got consent.
Seriously, all the industry researchers at these companies went through IRB training during their PhD; have they just conveniently forgotten?
Am I the only one who finds Claude overly assertive and confident for research? I tend to get more value out of other models (e.g. Gemini 4 Argon) even if they make mistakes, and then use Claude to revise and polish.
They are very good research assistants.
I would never trust an AI to make a scientific conclusion, but they're very good at some things:
- Plotting with natural language: "Create a scatter plot of column A versus column B, log scale Y and use stars sized by column C." Every plotting library (except maybe ggplot2 / plotnind) sucks, and the boilerplate never gets less tedius.
- Literature search, e.g. "Have there been any peer reviewed papers where they applied X method to Y domain"
- Complex visualization, e.g. "Make a self contained HTML file with a 3D interactive visualization of this graph. Add ball and spring position optimization and let me move nodes"
- Trying random stuff that would normally be too time consuming to be worth it, eg "Here is a dataset. Try regression methods A, B, and C and report a table." In this case, usually it will screw up and overfit, but it's a good smoke test.
Note how in all of these, it's being used for tedious things that provide some value, but ultimately they are only providing information to make a decision on.
Granted, I would like to have a research assistant I could train, but if your company will not create a position then there is not much you can do.
Yes to all but I have trouble using them effectively for my research side projects or Kaggle competitions. Claude Code/Codex designs and runs experiments quickly, invalidates my initial idea using some obscure criteria than changes direction, and after long hours and days of experiments we end up with poor results. There must be a better agentic workflow for research.
> invalidates my initial idea using some obscure criteria than changes direction
You might be giving it too much leeway.
Opus and Fable are more trustworthy, but even today I find they very quickly go off the rails for research tasks if I don't give them short chunks of work.
They're not ready for prime time automation of research (and before someone mentions the math results, that required absurd amounts of computation time and has immediate feedback in the form of Lean)
> Literature search, e.g. "Have there been any peer reviewed papers where they applied X method to Y domain"
This is something the LLMs are really good for, at least for compiler papers.
They are great computer interface assistants, but forget scientific conclusion would never trust then for committing in right git branch
This article talks about the outputs being Claude artifacts. I am curious if anyone has feedback on using Claude artifacts? What do you like/not like?
As a hobby I run https://smalldocs.org / https://github.com/espressoplease/smalldocs an agent agnositc artifact layer, so I'm looking for feedback on artifact use in general.
Here are some sample scientific SmallDoc artifacts:
Palmer penguins: body mass, flipper length and species - https://smalldocs.org/s/YrJDCfCD8-6j_uRE0pNCbI#k=odJRxWOIySV...
AlphaFold 3: capabilities, evidence and operational limits - https://smalldocs.org/s/rRRuByQhmhZNBH_4RNCx1t#k=S0Jhyf72Q6M...
AI in science is a lot different than AI in progamming in some ways. It seems pretty harmless in programming a lot of the times because it's used to speed up rote tasks like writing boilerplate. But in science, if it does give scientists a boost in discovery, I think it's likely to lead to an even more cavalier attitude about the power that such discoveries can have, and thus lead to much more abuse of power by the wealthy and malicious actors.
> Using privacy-preserving methods, we study 207,000 Claude and Claude Code research sessions to understand how Claude is used across scientific fields and research tasks.
Did they get permission to use researchers' data or did they monitor and extract data without consent? Considering the kerfuffle OpenAI generated with their math proofs, you'd think Anthropic would take a hint and be clear that they did the research with consent, though that assumes they got consent.
Seriously, all the industry researchers at these companies went through IRB training during their PhD; have they just conveniently forgotten?
https://arxiv.org/abs/2412.13678
Isn’t there something in the terms of service that gives them essentially blanket permission to read you conversations?
Am I the only one who finds Claude overly assertive and confident for research? I tend to get more value out of other models (e.g. Gemini 4 Argon) even if they make mistakes, and then use Claude to revise and polish.