Why is yield memory efficient?

Generators can be a difficult concept to get used to since Python is very "abstracted" and we are encouraged to write clean code. We rarely think about how much memory is allocated when we begin instantiating objects nor about the run-time efficiency. I rarely write my original code using generators (first principles: keep it simple, make it work), and "yield" comes in when I am refactoring or refining code to increase its speed.

A generator function saves time since it can begin returning results while it is still "running" rather than requiring an entire sequence to fully complete. It can save memory because if "yield" is applied properly, it should not have to build a completely new list of instantiated objects.

I came across this blog post which we discussed at a local Python meetup a couple of years ago.

"Python generators – saving time and memory" by Scott Moonen

https://scottmoonen.com/2008/02/01/python-generators-saving-time-and-memory/

Moonen's post compares creating a web page with a normal list build "get_objects()" function (using append to build and return a list result) to another build of get_objects() using a generator (yield):

def get_objects() :
  result = []
  query = db_execute("...")
  row = query.fetchrow()
  while not row is None :
    result.append(MyObject(row))  # Build object from DB, append to result
    row = query.fetchrow()
  return result
. . .
for object in get_objects() :
  print object.getHTML()

"This code creates the entire list of objects before it can print any of them. There are two problems with this — first, there is a huge delay to create all of the objects before any progress is made in printing; this means that the user’s browser has no partial results to display. Second, all of the objects must be held in memory at the same time; if there are many objects, this can cause significant overhead.

Generators allow us to attack both of these problems. Since a generator produces items one at a time, on demand, it avoids both of these problems. We don’t have to wait to construct all of the objects in the list before we use the first one. And once we are done using an object, the Python garbage collector is now free to immediately clean it up."

def get_objects() :
  query = db_execute("...")
  row = query.fetchrow()
  while not row is None :
    yield MyObject(row)    # Build object from DB, yield to caller
    row = query.fetchrow()
. . .
for object in get_objects() :
  print object.getHTML()

I hope this helps. Good luck with your Python journey!