Neum AI

Neum AI is an open-source framework designed to streamline the creation and management of data pipelines for Retrieval Augmented Generation (RAG) and semantic search. It focuses on transforming raw data into structured, embeddable representations that can be queried in real-time to provide context-aware responses within AI applications. The framework emphasizes large-scale and real-time data processing, enabling developers to bring up-to-date information into their AI systems without building custom infrastructure from scratch. At its core, Neum AI provides a set of open-source Software Development Kits (SDKs) that allow users to compose data flows programmatically. These SDKs handle key transformations such as data loading, chunking, embedding, and storing. The loading step ingests data from various sources—websites, databases, documents, APIs, and more—using built-in connectors. The chunking process splits large documents into manageable segments based on configurable strategies (e.g., sentence boundaries, token limits, overlap), ensuring that downstream embeddings capture meaningful context. Embedding then converts these chunks into numerical vectors using popular models like OpenAI, Cohere, or sentence-transformers, which are subsequently stored in a vector database for efficient similarity search. Neum AI supports a modular connector ecosystem. It includes pre-built connectors for common data sources (e.g., file systems, cloud storage, content management systems), embedding models (e.g., Hugging Face, OpenAI), and vector databases (e.g., Pinecone, Weaviate, Chroma). Users can also extend the framework by writing custom connectors using the same open-source architecture, allowing integration with proprietary or niche services. This modularity reduces the effort need