Scientific computing · Machine learning and data pipelines
Machine learning and data pipelines
Models are only as dependable as the data feeding them and the checks around them. This area covers machine learning for physical and scientific data, and the data pipelines, scheduled workflows and retrieval-based AI tools that keep results tested and reproducible.
Many organisations hold useful data in spreadsheets, exports and separate systems, and want to use machine learning and AI tools on it without losing control of accuracy. The work here builds pipelines that collect, clean and check that data, and adds learned models only where they help, with answers drawn from the organisation's own records and measured before anyone relies on them.
Machine learning for physical and scientific data
- Surrogate models. A Gaussian-process emulator learns the correction to a cheap formula from a small number of full simulations, and its own uncertainty is checked against the exact answer. The method demonstration reports where that interval fails to cover, not only where it works. See the work.
- Bayesian calibration from measurements. CASE / 001 infers a solar module's temperature behaviour from public measurement data and states how tightly the data constrain it. The data are fetched by script and filtered in code, so the study reruns from the raw download. See the work.
- Statistical learning with error estimates. Regression and classification, regularisation, neural networks, decision trees, support vector machines and dimensionality reduction, with errors estimated by cross-validation, bootstrapping and blocking. This is the toolkit behind the models above, learned on the Erasmus+ course Data Analysis and Machine Learning. Course material.
These are method demonstrations and one public-data study, not client projects. The machine learning projects further down come from a training programme and work on non-physical data.
What this covers
- Machine learning. Models for physical and scientific data with their uncertainty stated, and retrieval-based assistants for organisational records. Example.
- Data pipelines. Loading data from files, databases and APIs, cleaning and validating it, and storing it in PostgreSQL or MongoDB. Example.
- Workflow automation. Scheduled workflows in Airflow that feed dashboards and send notifications. Example.
- AI integration. Assistants that answer from an organisation's own documents or listings through retrieval-augmented generation, not from a language model's general knowledge. Example.
- Data assessment. Judging whether existing data is complete and reliable enough to support a decision. Example.
How the work is done
- Tested. Data-quality and integrity checks run inside the pipeline, alongside automated tests in pytest.
- Containerised. Each part runs in Docker, so the same pipeline runs on a laptop, a server or a cloud platform.
- Reproducible. Code is kept in Git with pinned dependencies, so every result can be rebuilt and checked.
- AI within clear limits. Answers come from retrieved sources, quality is measured on a fixed set of test questions before use, and what the system does not cover is written down.
Programme projects
The projects below were built on case studies set by a training programme. They are proofs of concept, not client work, and no client results are claimed.
- 01 /
Employee sports benefits data platform
Problem. A company wanted to test the feasibility of two employee benefits and estimate their financial impact.
Approach. An end-to-end pipeline that loads and validates HR and activity data, orchestrates the workflow with Airflow, posts activities to Slack and presents indicators on a Superset dashboard. PostgreSQL stores the data, numbered Bash scripts serve as a manual fallback, and everything runs under Docker Compose.
Validation. Data-quality checks run in the pipeline, and commute distances are validated with the Google Maps Distance Matrix API.
Outcome. A proof of concept. The data are simulated (161 employees); activity data is not read from a live service.
- 02 /
Healthcare data migration pipeline
Problem. A healthcare provider with scalability problems needed its medical records migrated from a CSV file to MongoDB.
Approach. A seven-step ETL workflow in Python, pandas and PyMongo with automated cleaning and integrity testing, a containerised deployment, and a written study of cloud deployment options on AWS.
Validation. Five integrity tests pass under pytest. 54,966 records remain after cleaning, from 55,500 in the source file. Categorical and datetime types cut memory use by 21%.
Outcome. A Docker Compose deployment of MongoDB with a web interface for the database, and an AWS comparison of DocumentDB, Atlas, EC2 and ECS.
- 03 /
Retrieval-augmented chatbot for cultural events
Problem. Recommending cultural events in Paris so that answers are grounded in real listings, not in a language model's general knowledge.
Approach. Events from the past 12 months in the Paris region are fetched from the Open Agenda API, cleaned, split into one text chunk per event, embedded and indexed with FAISS for similarity search. One LangChain chain serves both the Streamlit chatbot and the evaluation.
Validation. Evaluation with RAGAS on 30 questions, and 25 automated pytest tests.
Outcome. A proof of concept: single-turn, one data source, manual rebuilds.
- 04 /
EduGlobe Insights: education data assessment
Problem. Judging whether World Bank EdStats data can support the international expansion of an online education service.
Approach. Exploration, cleaning and analysis of the five EdStats tables in three Jupyter notebooks (pandas, Matplotlib, Seaborn, Plotly and SciPy). The tables are downloaded separately and are not stored in the repository.
Validation. Indicators are assessed for relevance, completeness and reliability, including the share of missing values by year.
Outcome. An assessment of which indicators and which years the data can support.
Early work
Repositories at an early stage. They are listed so the work is visible, and they report no results yet.
Climate time-series analysis
Daily Delhi climate records (mean temperature, humidity, wind speed and mean pressure, from January 2013), supplied as separate training and test files, with the packaging for a Python project around them. The analysis and models are not written yet.
Machine learning project revision
A revision of a Master's machine learning project: notes on supervised learning, loss functions and optimisation, ordinary least squares against ridge regression and the main families of neural network, with an introductory notebook and a linear regression script. Study notes and exercises; no results are published.
For enquiries about machine learning, pipelines or AI integration, write via Contact.