The washers are normally metal. If you pop rivet soft material then as like as not the end of the pop rivet will just pull back through the material as it's compressed leaving a messy hole but no bond. The washer spreads the load across a larger area and gives a stronger grip.