Data Engineering · BI · Automation

Sokoni Insights — the weekly report that runs itself.

A scheduled ETL pipeline that pulls from multiple sources, cleans and normalises the data, and publishes a live dashboard the team checks daily. Manual reporting disappears — and everyone works from the same numbers.

Read the case study Build a data pipeline

6 hrsManual reporting removed / week

4Sources unified into one model

DailyFreshness, down from weekly

sokoni — operations overview LIVE

Revenue · MTD

KES 2.41M

▲ 12.4%

Orders · MTD

1,284

▲ 8.1%

Return rate

3.2%

▼ 0.6%

Weekly revenue

last 8 weeks

W1W2W3W4 W5W6W7W8

POS → M-Pesa → Sheets → Warehouse ✓ synced 04:12

The problem

Four sources. One spreadsheet. Zero trust.

The business ran on a weekly ritual: export from the POS, export from M-Pesa, pull the ops sheet, paste everything into one workbook, and hand-build the same charts again. The numbers were always a week stale — and two people could get two different totals for the same week.

Before

~6 hours a week rebuilding the same report by hand
Numbers a week old — decisions made on stale data
Every metric defined slightly differently each time

After

Pipeline runs nightly and on demand — zero manual work
Dashboard refreshed daily, same numbers for everyone
Metrics defined once in a versioned data model

The pipeline

Extract, validate, model, publish.

Each stage is independently runnable and observable, so a failure at any point is caught, alerted, and recoverable without re-running the whole chain.

01 · EXTRACT

Pull from every source

Scheduled jobs fetch from APIs, databases and spreadsheets on an idempotent, date-windowed basis.

PythonRESTSQL

02 · VALIDATE

Catch bad data early

Schema, null-rate and range checks run before anything lands. Failures alert instead of silently propagating.

PanderaGreat Expectations

03 · MODEL

Define metrics once

Revenue, orders and returns are computed in a versioned model — so definitions can't drift between reports.

dbtPostgreSQL

04 · PUBLISH

Refresh the dashboard

The BI layer reads from the model, so every chart and KPI on the dashboard traces to the same source of truth.

MetabaseSQL

            
            dags/sokoni_daily.py
          
# Nightly pipeline — extract, validate, model, publish
with DAG('sokoni_daily', schedule='0 3 * * *', catchup=False) as dag:

    extract = PythonOperator(
        task_id='extract_sources',
        python_callable=pull_all_sources,   # POS, M-Pesa, ops sheet
        retries=3, retry_delay=timedelta(minutes=5),
    )

    validate = PythonOperator(
        task_id='validate_contracts',
        python_callable=run_quality_checks,  # schema, nulls, ranges
    )

    model = BashOperator(
        task_id='dbt_run',
        bash_command='dbt run --select marts.sokoni+',
    )

    publish = PythonOperator(
        task_id='refresh_bi',
        python_callable=refresh_dashboard,
    )

    extract >> validate >> model >> publish

Data quality

The checks that keep the numbers honest.

Every load is scored against a set of expectations. When something breaks, the pipeline alerts and halts rather than publishing numbers nobody should trust.

Quality checks · last run 03:04 EAT

All passing

Check	Table	Expectation	Result	Status
Row count anomaly	fct_orders	±15% vs 7-day avg	+4.2%	Pass
Null revenue	fct_orders	0%	0.00%	Pass
Payment reconciliation	fct_payments	≥99.5% matched	99.7%	Pass
Freshness	stg_pos_sales	< 24h old	6h 12m	Pass
Duplicate order IDs	stg_orders	0	2 flagged	Review

Capabilities

What the platform gives you.

Unified data model

Four sources joined into clean fact and dimension tables — so "revenue" means the same thing in every chart on every page.

Live BI dashboard

Revenue, orders, returns and channel performance — refreshed daily, filterable by period, and readable on a phone.

Automated validation

Contract-based checks run on every load. Bad data triggers an alert and blocks the publish — it never reaches a decision-maker.

Failure alerting

Task failures, SLA breaches and unexpected drops post to Slack with the failing step and a link to the logs.

Backfills & replay

Historical windows can be re-run safely thanks to idempotent writes — useful when a source corrects past data.

Documented & versioned

Transformations live in version control with tests and docs, so the logic behind any number is explainable months later.

Results

Time back, and numbers people trust.

6 hrsManual reporting time removed every week

DailyData freshness, up from a weekly snapshot

1 sourceOf truth for every reported metric

99.7%Payment-to-order reconciliation rate

“Monday used to be report day. Now the dashboard is just... there, and it's already right.”

— Operations manager, retail deployment

Stack

Built with.

Python Airflow dbt PostgreSQL Pandas Great Expectations Metabase Docker AWS

Still building the same report every week?

I build pipelines and dashboards that run on their own — with validation strong enough that you can actually trust what you're looking at.

Start a conversation Back to portfolio