AI in Data Science

Introduction

Large Language Models (LLM’s) have been under development for a long time. There has been a lot of progress since the first AI, ELIZA in the 1964! We have been using AI in the form of machine learning (ML) approaches in biology and business for many years already.

Development of AI and ML source
Note

Data science in biology and in business is not that fundamentally different, in my opinion. There are some different underlying principles that are unique to distinct dataset, but in general, a dataframe of purchasing habits is not different from a dataframe containing penguin dimensions. Therefore, I will use sources from business and science

The access to large amounts of data has enabled us to apply LLM’s in the way we currently do. Having minimal human interaction in the basic training of AI has allowed the algorithms to extract information and draw associations in a way we have been unable to do before!

Training cycle for LLM’s

The interaction of humans with LLM training is really important, and gives each LLM their unique traits. in April 2026, OpenAI released a statement about the increased mentions of goblins that seemed to have originated from an earlier model that had a “nerdy” personality users could select, and these trends emerged through transfer from the nerdy personality. Human involvement put a hard limit in the algorithm to ensure it would not mention goblins unless it was strictly necessary to the conversation with the user.

LLM’s are only as powerful as the data they are trained on. Data needs to be of high quality, and there needs to be enough data to train the algorithm. This is due to the tokenization process that underlies the basic functioning of the LLM’s we are currently using, assuming we aren’t using other tools along with the LLM.

Next token prediction of LLM’s source

It is also realluy important to remember that LLM’s don’t work in the same way as search engines, and should not be used like that without using appropriate tools/connectors.

Words of Caution

I have a whole document about using AI with care, thinking about what you are asking it to do, and what you are providing it. Please read through it!

As a general guideline

  1. Am I phrasing my prompt in a good way? Here is a guide to prompt engineering that might be useful
  2. Can I find this information any other way?
  3. How much time am I saving by looking this question up here vs on BioStars, for example?
  4. Do I know enough about the topic to know whether the GPT is lying to me?
  5. What are the consequences of testing the validity of the GPT solution? Can I potentially corrupt my data or my system? Is there a potential for me to lie to someone who trusts me enough to ask my opinion?

AI will generally hype you up and agree with you, so be cautious in how much you invest into it emotionally

Claude

There are many different LLM’s available online. My favourite is Claude from Anthropic. It has a fairly good data privacy policy, and the connectors available to users are also really cool!

Agents

Agents are advanced AI systems that implement the tools and memory from the LLM’s to perform tasks, call and/or develop tools, and reason to produce a final answer. There is currently a lot of debate about whether it is worthwhile for users to code their own agents, or not. In my opinion, Anthropic has a lot resources (intellectual, financial, physical) that I do not have access to, and they are developing excellent agents. In my opinion, the successful application of agents is more important to thoughtful AI use than novel development.

Connectors

There are many connectors that have already been developed for Claude by Anthropic, but you can also import custom connectors from external MCP servers. You can apply settings so that you only enable connectors when Claude calls on them, which makes the context window a lot smaller!

Skills

Claude has access to skills to extend Claude’s capabilities. There are many repositories for skills that you can import into Claude like this and this as examples. But always read through the skills you are enabling and downloading because there are a lot of bad actors in the AI space that want to hack your machine and steal your tokens and/or information.

These work a bit different you use Claude locally or only on the browser- see here for an overview.

Running LLM’s locally

You can run LLM’s locally through Ollama and LLM’s from Hugging Face. The advantages here are that you are able to run the LLM’s entirely offline manner. This is the ideal way to run these algorithms if you are working with any kind of sensitive data! You can also use LLM’s through R with ellmer if your analyses are limited to R.

Ollama logo

Additional Reading

https://toloka.ai/blog/the-backbone-of-large-language-models-understanding-training-datasets/ https://mlfrontiers.substack.com/p/a-friendly-introduction-to-large https://albanna-tutorials.com/llm_agents.html