A substantive test is when the auditor reperforms an operation, such as doing a bank reconciliation or checking expense coding on purchase invoices.
A compliance test, now usually called a test of control, is when the auditor checks that a control has operated. For example, seeing if the chief accountant has signed off the bank reconciliations as approval of them.