In part one of this series, we introduced CrewAI by mapping its features to the key attributes of an AI agent. Now, we will take a closer look at the core concepts that make CrewAI truly powerful and explore how its architecture enables developers to build sophisticated, intelligent systems.
At the core of CrewAI lies the concept of role-based AI agents. Each agent is designed to perform a specific task, with its behavior guided by defined roles, goals, and backstories. These agents don’t simply execute tasks — they dynamically collaborate, share information and adapt to the needs of the workflow.
Consider a scenario where an agent acts as a market research analyst tasked with identifying market trends. This agent can be configured as follows:
|
|
from crewai import Agent from crewai_tools import SerperDevTool
researcher = Agent( role=“Market Research Analyst”, goal=“Identify emerging market trends”, backstory=“An experienced analyst with a focus on technology and startups.”, llm=“gpt-4o-mini”, # Specifies the language model tools=[SerperDevTool()], # Integrates a web search tool memory=True, # Retains interaction history allow_delegation=False, # Restricts task delegation verbose=True # Enables detailed logs ) |
Here, the agent is configured with a specific role, clear goal and backstory to contextualize its behavior. Additional attributes, such as integrated tools (e.g., web search) and memory capabilities, make the agent intelligent and adaptable to the workflow.
Modularity in CrewAI
One of CrewAI’s most powerful features is its modular design, which allows developers to seamlessly associate various components — large language models (LLMs), tools, vector databases and memory — with agents. This modular architecture ensures that agents can be customized and extended to suit a wide range of tasks and workflows without significant reconfiguration.
CrewAI agents are LLM-agnostic, meaning they can leverage any open source or proprietary language model depending on the task requirements. Developers can specify the LLM provider and model at the agent level, ensuring flexibility in choosing models that align with performance, cost, or privacy needs.
For example, integrating OpenAI’s GPT-4o-mini or other models is straightforward:
|
|
from crewai import Agent, LLM
custom_llm = LLM( model=“gpt-4o-mini”, temperature=0.7, max_tokens=4000 )
researcher = Agent( role=“AI Researcher”, goal=“Analyze AI adoption trends”, backstory=“A data-driven analyst specializing in AI trends”, llm=custom_llm ) |
CrewAI supports vector database integration to enable retrieval-augmented generation (RAG) workflows. By combining language models with vector embeddings, agents can retrieve contextually relevant information from structured or unstructured data sources, improving response accuracy and grounding.
For instance, an agent configured to search a vector database for research content can be defined as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
|
from crewai import Agent, Task, Crew, tools from langchain.vectorstores import Chroma from langchain.embeddings import OpenAIEmbeddings
# Initialize vector database vectorstore = Chroma( embedding_function=OpenAIEmbeddings(), persist_directory=“./research_db” )
# Create a custom tool for vector search @tools.tool(‘Vector Search’) def vector_search(query: str) –> str: “””Search vector database for relevant context””” results = vectorstore.similarity_search(query) return str(results)
# Create a research agent with vector database access researcher = Agent( role=‘Research Specialist’, goal=‘Retrieve precise information from vector database’, backstory=‘I am an expert at retrieving and analyzing information from databases’, tools=[vector_search], verbose=True )
# Define task utilizing vector database research_task = Task( description=‘Conduct targeted research using vector database’, agent=researcher )
# Create crew crew = Crew( agents=[researcher], tasks=[research_task], verbose=True )
# Execute workflow result = crew.kickoff() |
Building Workflows With CrewAI
CrewAI facilitates seamless multi-agent coordination by enabling agents to work collaboratively within structured workflows. Workflows can be defined as sequential, hierarchical or asynchronous, depending on the nature of the tasks.
A simple example illustrates how two agents — a market researcher and a content strategist — collaborate to generate insights and develop a marketing strategy.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
|
from crewai import Agent, Task, Crew from langchain_openai import ChatOpenAI
researcher = Agent( role=‘Market Research Analyst’, goal=‘Discover emerging market trends’, backstory=‘Expert in identifying innovative business opportunities’, verbose=True, allow_delegation=False, llm=ChatOpenAI(model_name=“gpt-4”) )
writer = Agent( role=‘Content Strategist’, goal=‘Create compelling marketing narratives’, backstory=‘Skilled at transforming research into engaging content’, verbose=True, allow_delegation=False, llm=ChatOpenAI(model_name=“gpt-4”) )
research_task = Task( description=‘Analyze current market trends’, agent=researcher, expected_output=“A detailed analysis of current market trends” )
writing_task = Task( description=‘Develop marketing strategy based on research’, agent=writer, context=research_task.output, expected_output=“A comprehensive marketing strategy document” )
market_crew = Crew( agents=[researcher, writer], tasks=[research_task, writing_task], verbose=True )
result = market_crew.kickoff() print(result) |
In this example, tasks are executed sequentially. The market researcher first identifies emerging trends, and then the content strategist develops a strategy based on those insights. CrewAI orchestrates the agents’ coordination seamlessly, ensuring efficient task execution.
Advanced Workflow Management With CrewAI Flows
For more complex scenarios, CrewAI introduces Flows — a modular and event-driven approach to managing AI workflows. Flows allow developers to chain tasks dynamically, manage state and implement conditional logic for decision-making.
Consider a scenario where we generate the name of a random city and retrieve a fun fact about it using a Flow:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
|
from crewai.flow.flow import Flow, listen, start from litellm import completion
class ExampleFlow(Flow): model = “gpt-4o-mini”
@start() def generate_city(self): response = completion( model=self.model, messages=[ {“role”: “user”, “content”: “Return the name of a random city in the world.”}, ], ) random_city = response[“choices”][0][“message”][“content”] return random_city
@listen(generate_city) def generate_fun_fact(self, random_city): response = completion( model=self.model, messages=[ {“role”: “user”, “content”: f“Tell me a fun fact about {random_city}”}, ], ) fun_fact = response[“choices”][0][“message”][“content”] return fun_fact
flow = ExampleFlow() result = flow.kickoff() print(f“Generated fun fact: {result}”) |
In this example, the Flow class orchestrates the workflow. The @start() method generates the name of a random city, while the @listen() decorator triggers a follow-up method to retrieve a fun fact based on the city’s name. Flows simplify the creation of adaptive, event-driven workflows that respond dynamically to changing states or triggers.
Flows, when combined with existing primitives of the CrewAI, become a powerful orchestration framework for designing and building complex workflows.
Implementing RAG With CrewAI
CrewAI’s flexibility extends to implementing RAG, a powerful technique for enhancing AI systems by integrating external information retrieval into the generation process. CrewAI agents equipped with tools like PDFSearchTool or WebSearchTool can extract and process data from various sources to produce contextually accurate results.
Here’s an example where a specialized agent extracts key insights from a research paper using RAG:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
|
from crewai import Agent, Task, Crew from crewai_tools import PDFSearchTool
# Define RAG tool rag_tool = PDFSearchTool( pdf=‘research_paper.pdf’ )
# Define agent with RAG capabilities researcher = Agent( role=‘Research Analyst’, goal=‘Extract key insights from research documents’, backstory=‘Expert at analyzing complex academic papers’, tools=[rag_tool] )
# Define task for RAG research_task = Task( description=‘Analyze the research paper and summarize key findings’, agent=researcher, tools=[rag_tool], expected_output=‘A detailed summary of the key findings from the research paper’ )
# Create Crew to coordinate workflow research_crew = Crew( agents=[researcher], tasks=[research_task], process=‘sequential’ )
# Execute RAG workflow result = research_crew.kickoff() print(result) |
Scalability and Modularity
CrewAI’s modular architecture makes it ideal for scaling AI systems across industries. Agents, tasks and tools are reusable components that can be combined in various configurations to solve complex problems. Whether you’re automating workflows in finance, orchestrating research in academia, or managing customer queries in e-commerce, CrewAI provides a consistent, extensible framework.
For example, creating a reusable “Research Crew” allows the same agents to analyze different data sources with minimal reconfiguration:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
|
from crewai import Agent, Task, Crew from crewai_tools import PDFSearchTool
def create_research_crew(doc): tool = PDFSearchTool(pdf=doc) agent = Agent( role=“Research Analyst”, tools=[tool], goal=“Summarize document insights”, backstory=“I am an expert research analyst who specializes in extracting and summarizing key insights from documents.” ) task = Task( description=“Analyze and summarize content”, agent=agent, expected_output=“A comprehensive summary of the key insights from the document” ) return Crew(agents=[agent], tasks=[task], process=“sequential”)
# Execute for multiple documents for pdf in [“doc1.pdf”, “doc2.pdf”]: crew = create_research_crew(pdf) print(crew.kickoff()) |
This modularity ensures scalability without compromising on maintainability or performance.
CrewAI’s architecture extends far beyond static workflows by enabling intelligent, context-aware, and collaborative agents. Through advanced features such as memory, planning, event-driven Flows and RAG integration, CrewAI provides developers with the tools to tackle sophisticated, real-world challenges. By combining adaptability, scalability and modularity, CrewAI empowers teams to design AI systems that are both dynamic and reliable, unlocking new possibilities for automation and intelligent task execution.
YOUTUBE.COM/THENEWSTACK
Tech moves fast, don’t miss an episode. Subscribe to our YouTube
channel to stream all our podcasts, interviews, demos, and more.
