I am going to disagree with you on this one. It's more than depending on the person and how fast they can go.
If the company does not have enough ramp people to do the job then a flight waits.
If the ramp equipment is missing or not working this holds up things.
If weather enters the equation then that affects the operation.
If Rob the Ramper hurt his shoulder throwing bags but doesn't want to call off because he's at risk of losing his job, then that affects the operation.
If there is no one to drive a jetway to the aircraft then that affects the operation.
If the f/a's can't board passengers because they are still cleaning another airplane on the other side of the airport....guess what?
And the list goes on and on. It all comes down to proper tools, proper planning (up to date software, IT, etc) and enough staff to perform as humans not automatons.
Just imagine what a workforce that respects their leadership, works for a fair wage and feels respected could do!