Very often teams will have multiple components under the same application that they would like to develop & test independently as well.For a complex pipeline, we recommend creating the following projects
A production project - ex: Chatbot Production
Goal: Source of truth for production
All traces from production & CI tests are logged in this project.
We want to make sure that the schemas in this project always maps to what we see in production & doesn’t get scrambled by team members trying different variations offline.
One project per testable component - ex: Chatbot Retriever
Goal: Source of truth for development
All traces from offline evaluation runs & unit tests are pushed to this repo.
Prompts & datasets for a particular piece of the pipeline can be managed in this project.
We want to make sure as we try different things for a particular piece of our application that we can have a clear log of what we have tried & what worked. All of this can be tracked here.
A catch-all/random project - ex: Chatbot Random
Goal: Track random inspirations/ideas in one place
Anything that’s not going to be referred to actively in future development, but would be good to be tracked somewhere.