(not the guy from Mario Bros.)

(except Spotify did name it after Luigi guy they just don't use his image so they don't get sued by Nintento)

What is Luigi?

Luigi is a Python-based data pipeline tool
Built by Spotify engineers
System for managing a data workflow
Create steps as Python code for processing data
Create dependencies between steps
Run steps with CLI tool
Similar to Airflow by AirBnB

pip install luigi

Based on DAGs

Yeah, I use DAGs:

Directed
Acyclic
Graphs

Example: MARC

Downloading MARC
Splitting a MARC file up
Joining MARC files
Extract ISBNs from MARC

Example

Also: email notifications! retry failing stuff!

##### Example 3
                    ``` python [3-9|11-30]
                    # marc-hello.py
                    
                    class NightlySnapshot(luigi.ExternalTask):
                    
                        def output(self):
                            return luigi.RemoteTarget("nightly-snapshot.mrc", 
                                SFTPHOST, 
                                username=USERNAME,
                                password=PASSWORD)

class CleanSnapshot(luigi.Task):
                        def output(self):
                            return luigi.LocalTarget("cleaned-snapshot.mrc",
                                format=luigi.format.Nop) # binary

def requires(self):
                            return NightlySnapshot()

def run(self):
                            with self.input().open("r") as fh:
                                # open fh with PyMARC and do things to it
                                # do self.output().open('wb') (another fh)
                                # write to it

```

(not the guy from Mario Bros.)

What is Luigi?

Based on DAGs

Yeah, I use DAGs:

Example: MARC

Example

Later...

Sending emails and retries

Works witout Cenral Scheduler

Retries can be configured globally or per task

Thats it!

EM: jfournie@vcc.ca

TW: @robo_james

GH: @jamesrf