Brushing aids in breaking up biofilms, algae, water mold (even though you may avoid seeing these problems).
Eventually, the filter gets plugged though and may also have in order to become backwashed take away the waste.