No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
-
Updated
May 27, 2025 - Python
#
etl-pipeline
Here are 2,056 public repositories matching this topic...
Build data pipelines, the easy way 🛠️
python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest
-
Updated
Jun 6, 2023 - TypeScript
Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
-
Updated
May 28, 2025 - Java
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
python data-science machine-learning etl pandas orchestration data-engineering data-analysis software-engineering feature-engineering dataframe hacktoberfest dag lineage etl-framework etl-pipeline rag mlops llmops
-
Updated
May 30, 2025 - Jupyter Notebook
Implementing best practices for PySpark ETL jobs and applications.
-
Updated
Jan 1, 2023 - Python
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
infrastructure aws postgres data airflow cloudformation cassandra cluster aws-s3 aws-sdk data-warehouse data-engineering data-lake aws-ec2 postgresql-database data-modeling cassandra-database etl-pipeline data-engineering-pipeline airflow-operators
-
Updated
Aug 26, 2022 - Python
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
Enterprise-grade and API-first LLM workspace for unstructured documents, including data extraction, redaction, rights management, prompt playground, and more!
-
Updated
Jun 1, 2025 - TypeScript
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
python data-science machine-learning etl numpy pandas data-engineering data-platform software-engineering feature-engineering dataframe dag hamiltonian etl-framework hamilton featurization etl-pipeline stitch-fix
-
Updated
Jul 3, 2023 - Python
A Clojure high performance data processing system
-
Updated
May 13, 2025 - Clojure
A blazingly fast general purpose blockchain analytics engine specialized in systematic mev detection
-
Updated
Apr 29, 2025 - Rust
Integrate LLM in any pipeline - fit/predict pattern, JSON driven flows, and built in concurency support.
python ai concurrency ai-agents etl-pipeline llm llm-agent ai-agents-framework agentic-ai-development
-
Updated
Mar 10, 2025 - Python
A simplified, lightweight ETL Framework based on Apache Spark
-
Updated
Jan 24, 2024 - Scala
The Supabase of AI era. A modular, open-source backend for building AI-native software — designed for knowledge, not static data.
backend chatbot developer-tools knowledge-base data-ingestion etl-pipeline rag data-enhancement vector-database llm ai-native generative-ai agentic-ai supabase-alternative
-
Updated
May 30, 2025 - TypeScript
Fluent interface for (async) iterables
python data etl iterator python3 streams immutability data-engineering fluent-interface collections asyncio visitor-pattern lazy-evaluation threads iterable decorator-pattern method-chaining iterator-pattern etl-pipeline reverse-etl
-
Updated
Jun 1, 2025 - Python
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
docker big-data cassandra apache-spark data-storage postgresql data-engineering apache-kafka data-processing data-pipeline real-time-analytics containerization apache-zookeeper apache-airflow etl-pipeline
-
Updated
Feb 14, 2025 - Python
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
data-science sql sql-server etl data-warehouse sql-query datascience data-engineering data-analytics datawarehousing data-analysis sqlserver datawarehouse data-cleaning datalake data-warehousing etl-pipeline etl-job data-lakehouse medallion-architecture
-
Updated
Apr 23, 2025 - TSQL
A simple Spark-powered ETL framework that just works 🍺
data-science machine-learning framework scala big-data spark pipeline etl data-transformation data-engineering dataset data-analysis modularization setl etl-pipeline
-
Updated
May 6, 2025 - Scala
Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
-
Updated
May 30, 2025 - Go
This is a template you can use for your next data engineering portfolio project.
-
Updated
Sep 10, 2021
Improve this page
Add a description, image, and links to the etl-pipeline topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the etl-pipeline topic, visit your repo's landing page and select "manage topics."