From the Assignment Library, you can remove an assignment by selecting the ellipses in the furthest right column and selecting the “Delete” option from the drop-down menu.
Note: Assignments added to a course or assigned to students are unable to be deleted.
The assignment is removed from the Assignment Library, however users with access to native Salesforce have the ability to "undelete" the assignment (users without access can contact their administrator). This can be done by navigating to the assignment in Salesforce and unchecking the isDeleted checkbox.
You can also remove an assignment from the assignment editing page. After opening up the assignment you can delete it by clicking the “Delete Assignment” option on the right hand side of the screen under the Publish box.