Python - Generators

The yield statement in Python plays a crucial role in cooperative multitasking, a programming paradigm where tasks voluntarily yield control to a scheduler, allowing other tasks to run. This enables efficient concurrency without relying on threads or processes. Let's delve into the concept with an example:

import time

def task(name, times):
    for _ in range(times):
        print(f'Task {name} is running')
        yield  # Yield control to the scheduler
        time.sleep(1)

# Cooperative multitasking scheduler
def scheduler(tasks):
    while any(tasks):
        for task in tasks:
            try:
                next(task)
            except StopIteration:
                tasks.remove(task)

# Example: Running two tasks cooperatively
task1 = task('A', 3)
task2 = task('B', 4)

# Scheduler manages the tasks cooperatively
scheduler([task1, task2])

Task A is running
Task B is running
Task A is running
Task B is running
Task A is running
Task B is running
Task B is running

In this example, the task function represents a cooperative task that prints a message and yields control to the scheduler using the yield statement. The scheduler, represented by the scheduler function, iteratively calls the next function on each task, allowing them to run cooperatively.

The tasks run in a cooperative manner, with each task yielding control to the scheduler, which then switches to the next task. This cooperative multitasking approach is useful in scenarios where traditional concurrency mechanisms may be impractical or resource-intensive.

Creating a generator for an infinite sequence of prime numbers involves implementing the logic to generate primes dynamically. Here's an example:

def is_prime(num):
    """Check if a number is prime."""
    if num < 2:
        return False
    for i in range(2, int(num**0.5) + 1):
        if num % i == 0:
            return False
    return True

def prime_generator():
    """Generate an infinite sequence of prime numbers."""
    num = 2
    while True:
        if is_prime(num):
            yield num
        num += 1

# Example: Generating first 5 prime numbers
prime_gen = prime_generator()
primes = [next(prime_gen) for _ in range(5)]

[2, 3, 5, 7, 11]

In this example, the is_prime function checks if a given number is prime. The prime_generator function is a generator that yields an infinite sequence of prime numbers. It starts from 2 and increments the number, yielding each prime number it encounters.

The example demonstrates using the prime_generator to generate the first 5 prime numbers. You can continue to use the generator to obtain more prime numbers as needed.

Generators are particularly useful for processing large log files efficiently. They allow you to iterate over the file line by line, keeping memory usage low. Here's an example:

def process_log_file(file_path):
    """Process a large log file using a generator."""
    with open(file_path, 'r') as log_file:
        for line in log_file:
            # Process each log entry (replace this with your actual processing logic)
            processed_entry = process_log_entry(line)
            yield processed_entry

def process_log_entry(log_entry):
    """Example processing logic for each log entry."""
    # Replace this with your actual processing logic
    return log_entry.strip().upper()

# Example: Processing a log file and printing the first 5 entries
log_file_path = 'large_log_file.txt'
log_entries_generator = process_log_file(log_file_path)
first_5_entries = [next(log_entries_generator) for _ in range(5)]

['LOG ENTRY 1', 'LOG ENTRY 2', 'LOG ENTRY 3', 'LOG ENTRY 4', 'LOG ENTRY 5']

In this example, the process_log_file function is a generator that reads a large log file line by line. For each line, it calls the process_log_entry function, which represents your actual processing logic for each log entry.

The example demonstrates using the generator to process the first 5 log entries, but you can continue to iterate over the generator to process the entire log file efficiently.

The yield statement in the context of stateful generators is used to temporarily pause the generator's execution and yield a value. When the generator is later resumed, it continues execution from where it left off, maintaining its internal state. This allows generators to represent sequences with potentially infinite or dynamic lengths without consuming excessive memory.

Let's explore an example of a stateful generator using the Fibonacci sequence:

def fibonacci_generator():
    """Generate the Fibonacci sequence using a stateful generator."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Example: Generating the first 5 Fibonacci numbers
fibonacci_gen = fibonacci_generator()
first_5_fibonacci_numbers = [next(fibonacci_gen) for _ in range(5)]

[0, 1, 1, 2, 3]

In this example, the fibonacci_generator is a stateful generator that yields Fibonacci numbers indefinitely. The yield a statement pauses the generator, returning the current Fibonacci number (a), and then resumes from the same point in the next iteration.

The first_5_fibonacci_numbers list shows the first 5 numbers generated by the stateful generator.